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SYSTEMS AND METHODS FOR GENERATING SPHERICAL MOSAIC 

IMAGES 

The present application claims the benefit of United States Provisional 
application United States Provisional application 60/154,722 filed on September 20, 
5 1999 and entitled "Omnidirectional Mosaijcing Using Curved Mirrors" and United 
States Provisional application 60/170,392 filed on December 13, 1999 entitled "360 x 
360 Mosaics". 

The subject matter of this application was supported in part by a grant from 
DARPA ONR/ARPA MURI, Grant No. NOOO 14.95- 1-0601 and, therefore, the 
10 Uftited States government may have certain rights to the invention. 

Field of the Invention 

The present invention relates generally to digital imaging and more 
particularly relates to the acquisition and combination of a plurality of image 
segments to generate spherical mosaic image data, 

15 Background of the Invention 

The generation of high resolution, large field of view images is important in 
many areas, such as virtual reality, computer vision, robotic navigation and' the like. 
It is generally known that mosaic images can be computed by stitching together 
multiple images, where the individual images correspond to different views of the 

20 same scene. Generally, the different views have approximately the same viewpoint 
but correspond to different sections of the scene of interest. A variety of techniques 
for image mosaicing are known. Such techniques generally employ an imaging 
system that uses a conventional imaging lens to form an image of the scene. A 
problem with conventional lenses is that they are limited in their fields of view in 

25 both image directions. Therefore, to construct a panorama or a complete spherical 
image, a large number of images need to be captured and processed. In addition, to 
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capture a complete spherical view the imaging systems need to scan the scene in three 
dimensions without missing any substantial areas. This is difficult to accomplish 
using conventional camera systems. 

An alternate approach is to use a wide angle lens such as a fish eye lens. This 
5 approach reduces the number of images since the field of view captured in each 
image is significantly larger than in the case of conventional images. However, 
when a fish eye lens is used, the resolution of the image is uniformly distributed over 
the increased field of view and as a result, such systems are not well suited for high 
resolution mosaicing. 

10 Catadioptric imaging systems which use a combination of mirrors and lenses 

to form an image are also known. The use of mirrors provides the designer great 
flexibility in terms of manipulating the field of view and resolution characteristic of 
the imaging system. Catadioptric systems have shown promise for wide angle 
imaging. 

15 Despite their known benefits, known catadioptric imaging systems cannot 

capture a complete spherical view within a single image. The field of view is 
typically limited by two factors, namely, a blindspot that results from the mirror (or 
mirrors) seeing the camera itself and the finite extent of the mirror (or mirrors) that 
corresponds to a truncation of the field of view. Furthermore, as the field of view of 

20 a catadioptric system is increased, such as by increasing the extent of the imaging 

mirrors, the resolution of the image is naturally reduced. This is because in any given 
image detector (CCD, film, etc.) the number of picture elements is fixed and an 
increase in the field of view corresponds to each pixel subtending a larger solid angle 
with respect to the captured scene. 

25 Several variations of wide angle imaging systems are known. As shown in 

Figure 2, a first system includes an image sensor 202, an objective (imaging) lens 204 
and a curved (convex or concave) mirror 206. The curved mirror 206 can have a 
variety of shapes. In the case of a hyperboloidal mirror, if the entrance pupil of the 
imaging lens is placed at the external (far) focus F'l of the mirror 206, only the scene 

30 rays that approach the near focus F, of the mirror 206 are imaged by the system. In 
this case, the near focus of the mirror is the center of projection of the complete 
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catadioptric system. 

More compact catadioptric designs are possible using the concept of optical 
folding which uses multiple mirrors. As shown in Figure 3, the near focus F2 of the 
secondary mirror 302 is made to coincide with the far focus F\ of the primary mirror 
5 304. Then, if the entrance pupil P of the imaging lens 306 is placed at the far focus 
F'2 of secondary mirror 302, the imaging system captures only those scene rays that 
approach the near focus F, of the primary mirror 304, making it the center of 
projection of the complete imaging system. 

A variety of mirror configurations may be used is catadioptric systems, with 
10 the mirror shapes being hyperboloidal, ellipsoidal, and paraboloidal. If the 

requirement that the center of projection should be a single point is relaxed, a variety 
of other shapes can also be used, such as spherical and conical. In fact, the mirror 
profiles are not confined to first order or second order profiles but can be higher order 
polynomials that permit the manipulation of the resolution characteristics of the 
1 5 imaging system. 

Catadioptric imaging systems include one or more curved mirrors in the 
imaging system and the mirrors may be concave or convex. In the case of multiple 
mirrors, there is at least one curved mirror and one or more of the others could be 
planar mirrors. 

20 Another embodiment of a catadioptric imaging system is illustrated in Figure 

4. This system includes a parabolic mirror 401, a relay lens 404, an imaging lens 406 
and an image detector 408. The relay lens 404 is typically an achromatic lens with a 
short focal length. The distances between the relay lens 404, the imaging lens 406 
and the parabolic mirror 402 are chosen such that only principal rays of light that are 

25 orthographically reflected by the parabolic mirror are imaged by the system. These 
reflected rays correspond to scene rays that approach the focus of the parabola. 

Typically, the image captured using a wide angle catadioptric system looks 
like that illustrated in Figure 5. This image is donut shaped and has two circles that 
are of the limits of the imaging system's field of view. The first inner circle B' 

30 corresponds to the blindspot that is caused by the primary mirror being obstmcted in 
its view by other mirrors, the imaging lens, or detector. The second (outer) circle L' 
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of the field of view results from the finite extent of the mirrors. As can be seen, such 
a field of view is 360 degrees in one dimension and less than 180 degrees in the other. 
It should be noted that the imaging lens and detector need not necessary be used to 
image the complete field of view captured by the mirrors. Instead, they may be used 
to image a smaller section of the donut image in order to increases image resolution 
(lower the solid angle imaged by each pixel). 



The present invention provides systems and methods for using an imaging 
system to efficiently scan a complete spherical view of a scene. 

In accordance with the present invention, slice image data is acquired by 
receiving rays over a large field of view in a first direction and a small imaging width 
in a second direction. At least a portion of the received rays are parallel with each 
other with respect to the second direction. That portion of the rays which are parallel 
in the imaging width are directed upon an area of an imaging sensor. Preferably, the 
directing step includes interposing at least one mirror and at least one lens between a 
point where the rays are received and the imaging sensor. 

The method of acquiring slice image data provides for the large field of view 
to be approximately 360 degrees. In this case, the area of the imaging sensor 
receiving the rays is an annular ring having a radial width. 

The methods for acquiring slice image data can be used by a slice image 
camera. A slice camera generally includes an imaging sensor, at least one mirror 
receiving rays over a large field of view in a first direction and a small imaging width 
in a second direction, and at least one lens. The at least one lens receives at least a 
portion of the rays which are incident on the at least one mirror and directs only the 
portion of the rays which are parallel to each other in the second direction to an area 
of the imaging sensor. 

Numerous arrangements of mirrors and lenses can be employed in forming the 
slice image camera. For example, a conical mirror can be used in cooperation with a 
telecentric lens. In addition, multi-mirror configurations are also provided which can 
reduce the size of the slice camera. 



SUMMARY OF THE INVENTION 
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In accordance with a first method of generating a spherical mosaic image, a 
plurality of image sections are acquired about an axis of rotation. The image sections 
have a large field of view along the axis of rotation and a small field of view (non- 
parallel rays) or width (parallel rays) in a second direction. Consecutive images about 
5 the axis of rotation are then combined to form the spherical mosaic image. 

The images can take the form of image strips, which are formed from a 
narrow field of view of convergent rays, or image slices, which are formed from a 
narrow width of parallel rays. The image strip typically corresponds to about 180 
degrees or 360 degrees in one dimension of the field of view and only a few degrees 

1 0 in the second direction. 

If the rotational position is known, consecutive images can be concatenated 
using the known rotational positions to form a substantially continuous spherical 
image about the axis of rotation. 
: If the rotational position of the images is not known, the images must take the 

1 5^ form of overlapping strips. In this case, common features are identified within the 
overlapping regions of consecutive image strips and the strips are aligned 
accordingly. Preferably, a blending algorithm is used to improve image fidelity in the 
overlapping regions. 

An imaging system for acquiring an omnidirectional mosaic image in 

20 accordance with the present invention includes an image slice camera having a large 
field of view in a first direction and a small width in a second direction. The image 
slice camera receives parallel rays of light which are incident on the camera width 
and directs the parallel rays onto an area of an imaging sensor. A rotational drive 
system is operatively coupled to the image slice camera. The rotational drive system 

25 rotates the image slice camera along an axis of rotation which is substantially parallel 
to the first direction. A control unit is operatively coupled to both the rotational drive 
system and image slice camera. The control unit receives the image slice data from 
the image slice camera and records corresponding rotational positional information of 
the rotational drive system. The control unit acquires a plurality of image slices at 

30 known rotational positions about the axis of rotation and concatenates the image 

slices to form a spherical mosaic image data. An output device is generally coupled 
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to the control unit for displaying a spherical mosaic image from the spherical mosaic 
image data. The output device can take the form of a video output device or 
hardcopy, such as a printer. 

A slice camera can be formed with at least one mirror receiving the parallel 
S rays and directing the rays through the entrance pupil of at least one lens and onto an 
imaging sensor. The mirror can take several forms, such as a non-convex mirror 
section or a conical mirror section. 

Multi-mirror embodiments can also be employed with combinations of 
various mirror shapes that direct only the parallel rays incident on the camera to the 

10 imaging sensor. For example, a slice camera can be formed with a first mirror 

section receiving the parallel rays, a second mirror section and an imaging sensor. 
The second mirror is positioned to receive the parallel rays from the first mirror and 
direct the rays through an entrance pupil of a lens and onto an imaging sensor. The 
second niirror section can exhibit a near focus and a far focus. In this case the first 

15 mirror is positioned to project the received rays through the near focus and onto the 
second mirror section which directs the rays to the image sensor located at a position 
proximate the far focus. Preferably, the near focus is located in front of the first 
mirror section and the far focus is located behind the first mirror section. 

The imaging sensor for the system can be any light sensitive device such as a 

20 CCD imager, a CMOS imager or photographic film. The imaging sensor should have 
image pixels distributed in at least two dimensions (e.g., X-Y) and the parallel rays of 
the image sheets are directed onto a substantial portion of the sensor. The distribution 
of pixels on the imaging sensor can be distributed only within, or concentrated 
within, the portion where the rays are directed, such as in an annular ring pattern. 

25 This facillitates lower cost imaging sensors (less pixels used) or higher resolution 
imaging sensors (same number of pixels concentrated into the active portion of the 
sensor) being formed. In the case of a video system, the output can be continuously 
captured while the imaging system is rotated with respect to a convenient point of 
rotation. 

30 Since the image sections generally have 180 degrees or greater field of view 

in one dimension, a single rotation of 1 80 degrees or 360 degrees (about a single axis) 
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of the slice camera (or strip camera) can provide a complete spherical scan of the 
scene of interest. The rotation can be accomplished by mounting the slice camera on 
a tripod stand, a motorized rotary stage or by simply holding the imaging system by 
hand and rotating it around an axis. 



be acquired in a single rotation about the axis of rotation. For instance, two views 
can be presented as a single image where the two views are color coded. Then, using 
eye glasses with different color filters for the left and right images, the observer can 
10 perceive the depths of objects in the scene. Alternatively, the observer can wear a 
head-mounted display and the two images can be provided to the left and right 
displays of the head-mounted display. In this mosaic, the stereo baseline (in the case 
of the human visual system, this would be the distance between the eyes) which is 
determined by the radial offset of the camera from the axis of rotation remains 

15 substantially constant over the entire mosaic. 

Preferably, a mapping algorithm is provided that enables a user to generate 
perspective views from the onmidirectional mosaics. Such views can be used to 
navigate around the 360 degree x 360 degree field of view. A variety of existing 
mapping techniques can be used to map from spherical mosaics to perspective views. 

20 The captured stereoscopic data can be used to compute an onmidirectional 

(e.g., 360 degree x 360 degree) depth map of the scene. Hence, a robust estimate of 
the coordinates of each scene point can be computed from the captured data. 

The rotation of the imaging system and the computation of the 
omnidirectional data may be accomplished in real-time using high speed motors and 

25 high performance processors. This enables real-time systems where the information 
(be it regular mosaic, stereo mosaic or depth map) is computed substantially 
continuously and hence can vary with changes in the scene (such as moving objects). 

The present invention provides an onmidirectional brightness/color image of a 
scene, an omnidirectional stereoscopic image of a scene and an onmidirectional depth 

30 map of a scene using a single 1 80 degree or 360 degree rotation of a catadioptric 
imaging system. Though it is desired to compute complete omnidirectional visual 



5 



The .slice camera has a center of projection and the axis of rotation can be 
radially offset from tlie center of projection. In this case, stereoscopic image data can 
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data, the field of view does not have to be a complete sphere but can include small 
blindspots (for instance, at the poles of the sphere). In addition, to improve image 
resolution, it may be desirable to only form the mosaic over a portion of the spherical 
field of view. Hence, the term onmidirectional data as used herein means data 
corresponding to a substantial portion of a spherical view and data corresponding to 
' panoramic data over a substantial portion of a spherical view. 

Brief Description of the Drawings 
Figures lA through ID illustrated representations of omnidirectional data in a 
different coordinate systems; 

Figure 2 is a first embodiment of a catadioptric wide angle imaging system, 
known in the prior art; 

Figure 3 is an alternate embodiment of a catadioptric wide angle imaging 
system using a primary mirror and secondary mirror, known in the prior art; 

Figure 4 is an alternate embodiment of a catadioptric wide angle imaging 
system, known in the prior art; 

Figure S is a diagram illustrating an exemplary imaging area produced by a 
catadioptric imaging system known in the art; 

Figures 6A and 6B are perspective views illustrating a sphericeil imaging area 
generated by combining a small number of wide angle images from a catadioptric 
imaging system; 

Figure 7A is a perspective view of an image "strip'*; 
Figure 7B is a graph illustrating the mapping of an image "strip" of the 
spherical image of Figure 7 A onto a plane; 

Figures 8A and 8B illustrate that mapping of an image "slice" of a spherical 
image onto a plane, respectively; 

Figure 9 is a schematic diagram of a catadioptric imaging system used for the 
acquisition of image strips in connection with the practice of the present spherical 
mosaicing imaging method; 

Figure 10 is a schematic diagram of an alternate embodiment of a catadioptric 
imaging system used for the acquisition of image strips in connection with the 
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practice of the present spherical mosaicing imaging method; 

Figure 1 1 is a schematic diagram of an alternate embodiment of a catadioptric 
imaging system used for the acquisition of image strips in connection with the 
practice of the present spherical mosaicing imaging method; 
5 Figure 12 is a graph illustrating the operating principle of acquiring an image 

"slice" in accordance witii tiie present invention; 

Figure 1 3 is a graph illustrating the application of a "hat shaped" mirror used 
for acquiring an image "slice" in accordance with the present invention; 

Figure 14 is a graph illustrating the application of a conic mirror used for 
10 acquiring an image "slice" in accordance with the present invention; 

Figure 15 is a graphical representation illustrating the mapping of image slice 
data points onto a region of an image sensor; 

Figure 16A is an intensity graph illustrating the data points acquired by an 
image strip along the outer circle of the graph in Figure 15; 
. 15 Figure 16 B is an intensity graph illustrating the data points acquired by an 

image slice, which is mapped into the region between the inner circle and the outer 
circle of the graph in Figure 15; 

Figure 1 7 is a first embodiment of a catadioptric imaging system for acquiring 
image "slices" using a conical mirror in accordance with the present mvention; 
20 Figure 1 8 is an alternate embodiment of a catadioptric imaging system for 

acquiring image "slices" using a non-convex mirror in accordance with the present 
invention; 

Figure 19 is an alternate embodiment of a catadioptric imaging system for 
acquiring image "slices" using a stacked mirror in accordance with the present 
25 invention; 

Figures 20A and 20B are alternate embodiments of catadioptric imaging 
systems for acquiring image "slices" using multiple mirrors in accordance with the 
present invention; 

Figure 21 is an embodiment of a system for acquiring image slices using a line 
30 detector; 

Figure 22 is a block diagram of an imaging system in accordance with the 
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present invention and illustrating the rotation of a slice camera offset from the axis of 
rotation to acquire stereoscopic image data in a single rotation; 

Figure 23 is a flow chart illustrating a method of generating a spherical 
mosaic using overlapping strip images; 

Figure 24 is a flow chart illustrating a method of generating a spherical 
mosaic using slice images or non-overlapping strip images; 

Figures 25 A and 25B are diagrams illustrating the mapping of image data to 
the spherical mosaic; 

Figure 26 is a perspective diagram illustrating the projection of a portion of 
the spherical mosaic into a plane to generate a perspective image; 

Figures 27A-C are top view diagrams illustrating the acquisition of left and 
right side images as an imaging system is rotated about a point which is radially 
offset from the center of projection; 

Figure 28 A is a perspective view illustrating exemplary changes in 
observation point a hum2in viewer can exhibit; 

Figure 28B is a perspective view illustrating the generation of left and right 
perspective views from left and right mosaics, respectively; 

Figures 29A and 29B illustrate an exemplary point in a scene as viewed from 
the left and right mosaics, respectively; 

Figure 29C is a perspective model illustrating the calculation of the depth of a 
scene point; 

Figure 30 illustrates the displacement of a point in the scene over time in a 
series of acquired images; 

Figure 3 1 is a perspective model illustrating the calculation of the depth of a 
scene point using the plurality of images of Figure 30; 

Figure 32 is an exemplary slice image acquired by a slice camera in 
accordance with the present invention; 

Figures 33A and 33B are left and right spherical mosaic images, respectively, 
formed in accordance with the present invention; and 

Figures 34A and 34B are left and right perspective images, respectively, 
formed in accordance with the present invention. 
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Detailed Description of Preferred Embodiments 

The present systems and methods are directed to the generation of 
omnidirectional mosaic images using acquired image "strips" or image "slices." 
Referring to Figure 7A, an image strip is a section of the spherical image having a 
5 narrow field of view which is generated by converging rays incident on the imaging 
system being directed onto an imaging surface. Figure 7B illustrates the mapping of 
the spherical image strip onto the 9-(|) plane. As used herein, an image slice 800, 
which is graphically illustrated in Figure 8A, is formed by parallel rays incident on 
the imaging system being acquired by an image sensor over a thin width of the 

10 spherical image. As illustrated in Figure SB, because the image slice is acquired from 
a narrow width of parallel rays, the image slice maps into a straight line 802 in the 6- 
(j) plane. It will be appreciated that the term rays as used herein is not limited to 
radiation in the visible light spectrum but generally refers to high-frequency 
electromaignetic radiation, including but not limited to visible light, infrared radiation, 

15 ultraviolet radiation, x-ray radiation and the like. 

As used herein, the term image section will be used to refer to both image 
slices and image strips. Further, the term "camera" will be used to refer to a photo 
imaging subsystem, from the point rays are incident on the sub-system through to the 
imaging detector, regardless of the nature of the optics interposed therein. A slice 

20 camera refers to a photo imaging subsystem for acquiring image slice data from 
incident parallel rays, and a strip camera refers to a photo imaging subsystem for 
acquiring image strip data from incident rays. 

Although the present invention does provide for full 360 x 360 spherical 
mosaic imaging, it is not limited to applications vnth complete spherical mosaics. 

25 Accordingly, the term omnidirectional mosaic and omnidirectional data as used 

herein refers to images and data corresponding to a substantial portion of a spherical 
view and/or data corresponding to panoramic data over a substantial portion of a 
sphere. 

To compute omnidirectional visual information, a convenient representation 
30 or data structure must be chosen. The omnidirectional data may be represented in a 
variety of ways, some of which are shovm in Figure 1 . Each point can be represented 
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by the polar angles (6, (\>). The resulting data may be stored in the form of (a) a 
discretized sphere, (b) a discretized cube or cuboid, (c) a discretized cylinder with top 
and bottom faces or in the form of (d) a spherical panorama that is a two-dimensional 
representation of the (6, <|>) space. The exact representation is not critical to the 
5 imaging techniques and mosaicing algorithms presented assuming that all stored 
points, irrespective of the representation, can be mapped to their (0, <|)) coordinates. 

Figure 6A illustrates the field of view of a typical catadioptric imaging system 
on a unit sphere. The circle B on the sphere corresponds to the blindspot and the 
circle L is the outer limit of the field of view. As shown in Figure 6B, three 

1 0 overlapping images from a conventional catadioptric imaging system may be 

sufficient to capture the complete sphere by using the orientations (directions of the 
optical axis, of the imaging system) O,, O2 and O3. 

The overlapping image regions will generally differ from each other slightly 
due to inaccuracies in the imaging system, illumination changes during image 

1 5 capture, and variations in the exposures of the three images. To handle such 

variations, a blending algorithm can be used to calculate the brightness and or color at 
each pixel in an overlap region of the spherical field of view as a weighted 
combination of the pixel's brightness and or color in the individual view. The 
weights can be the shortest distances between the pixel under consideration and the 

20 boundaries of the individual fields of view. This is illustrated as dl, d2, and d3 in 
Figure 6B. Therefore, the brightness at the point shovwi in Figure 6B may be 
calculated using the known equation: 

For a typical catadioptric imaging system, a minimum of three views would be 
25 needed to capture the complete sphere of view. 



Preferably, the spherical mosaic is generated using image sections which are 
acquired in a single rotation of a camera about a single axis of rotation. In the case 
of an image strip, the field of view of the imaging system is reduced to only a few 
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degrees in one dimension. For example, a strip can be 360 degrees in one dimension 
and of relatively small field of view in the other dimension. In this case, a rotation of 
1 80 degrees is sufficient to capture the entire region. Alternatively, the strip can 
exhibit 1 80 degrees in one dimensions and again a relatively small field of view in the 
5 other dimension. In this latter case, the image resolution is improved and a 360 x 360 
image is acquired by a full 360 degree rotation of the imaging camera. 

Rotation of the catadioptric imaging camera can be accomplished in a number 
of ways. For example, the system can be rotated using a motorized stage with a 
single axis of rotation. In this case, the rotation between images can be precisely 

10 determined and performed automatically via computer control. Alternatively, the 
imaging camera can be mounted on a tripod and manually rotated about the axis of 
rotation. In this case, the rotational position of the cameria may be known, such as by 
using a sensor or by reading from a rotation dial on the tripod, or it may be unknown. 
Alternatively, the system may be rotated approximately about its center of projection 

15 by a human by simply holding it in his/her hand. In this case, the rotation between 
consecutive images is unknown and must be automatically recovered from the 
captured images. 



20 mirror 902 which is placed in front of an imaging lens 904 that directs the convergent 
rays of the strip onto an imaging sensor 906, such as a two-dimensional CCD 
detector. An advantage of this system is that the resolution of the image of strip can 
be significantly greater than in the case of a pure lens systems such as a fish eye lens. 
This arises from the fact that in the case of curved mirrors, the catadioptric image 

25 increases in resolution approaching the edge (boundary L) of the field of view. It has 
been shown that the resolution of the imaging system is: 



Figure 9 shows an embodiment of a catadioptric imaging system that produces 
a thin 360 degree strip of the field of view. The system includes a section of a curved 



dA 




dA 




(2) 



where c is the distance between the near and far foci of the mirror 902, which is also 



wo 01/22728 PCT/USOO/25801 

14 

the distance between the center of projection and the entrance pupil 908 of the 
imaging lens 904. Therefore, the resolution of the complete imaging system increases 
with r and is maximum at the periphery of the field of view (which corresponds to the 
boundary L on the sphere). Therefore, unlike a fish eye lens, the catadioptric system 
5 of Figure 9 captures an image strip that has highest resolution within the 1 80 degree 
or 360 degree strip. 

An alternate embodiment of a catadioptric imaging system is shown in Figure 
10. Referring to Figure 10, the rays from within the field of view 1000 of the strip 
are directed onto a section of a parabolic mirror 1002 and to an image sensor 1008. 
10 The mirror 1002 directs the rays onto a relay lens 1004 which focuses the rays 

through an objective lens 1006 and onto the imaging sensor 1008. In this case, the 
resolution of the catadioptric system is known to be: 

dA / ^ -yxdA 



where, z(r) is the equation of the profile of the parabola: 
.5 z.— (41 

where, h is the parameter of the parabola. For instance, the resolution of the image 
for scene rays that are 90 degrees with respect to the optical axis is twice that for the 
scene ray that is along the optical axis. Hence, the thin strip is guaranteed to have the 
maximum image resolution. 

20 Thin-strip catadioptric imaging systems can also be constructed using multiple 

mirrors. Multiple mirrors can further reduce packaging size as well as compensate 
for the slight field curvature that remains in a single mirror system. Such a system is 
shown in Figure 1 1. A convex primary mirror 1 102 and a concave secondary mirror 
1 104 can be used to introduce field curvatures in opposing directions that tend to 

25 cancel the effects of each other as the rays are directed to the imaging sensor 1 106. 
As in the conventional systems, if the mirrors have conic sections, they can be 
arranged so as to ensure that the center of projection of the imaging system is the near 
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focus of the primary mirror. 

Thin-strip cameras can be constructed from a variety of mirrors. The mirrors 
can be parabolic, hyperbolic, ellipsoidal, conical, spherical or planar. Alternatively, 
the mirrors can be higher order surfaces that are used to achieve greater resolution 
5 within the strip. When the mirrors are not parabolic, hyperbolic or ellipsoidal, the 
center projection is most likely not a single point but rather a locus of viewpoints. In 
such cases, the locus is expected to be compact enough to be approximated by a 
single point. 

The acquired image sections are combined to form a resulting omnidirectional 
10 mosaic image. If the camera rotation is performed in a controlled manner such that 
the rotational position of each image section is known, the image sections or slices 
can simply be concatenated to form a spherical image. In those cases where the 
rotational position is not known, feature matching can be performed in overlapping 
strip regions to identify common features. These conunon features are then used as 
15 reference points to combine the overlapping image strips. In the region of overlap, a 
blending algorithm can be employed to restore the color and intensity of the image 
data. 

While catadioptric imaging systems capable of acquiring image strips are 
known, this is not the case with the acquisition of an image slice. As noted above, in 

20 the case of the image slice, only rays from the source which are parallel to each other 
with respect to the direction of the slice thickness and are incident on the camera are 
detected. Thus, the catadioptric imaging system must be modified such that only 
those parallel rays of the image slice thickness are directed to the entrance pupil of an 
imaging lens and then onto as imaging detector. Figure 12 graphically illustrates a 

25 mathematical model for deriving various mirror profiles which can achieve this 
result. In Figure 12, a cross section of mirror 1202 is illustrated. In practice, this 
mirror circularly extends around a substantial portion of axis z to establish a wide 
field of view in a first direction. This wide field of view can be 360 degrees around 
the axis or a substantial portion or portions thereof The parallel sheet of rays are 

30 shown incident on the mirror 1202 substantially parallel to the image plane 1204 
(P+Y =90). In this arrangement, the sheet of rays, which are parallel in a direction 
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sheet. Analysis of the model of Figure 12 yields an equation for the mirror profile 
wherep+Y =90of : 



results in the "hat-shaped" profile illustrated in Figure 13. In the case of orthographic 
projection, the angle P (Figure 12) is 7i:/4 and the resulting mirror profile is a cone, as 
illustrated in Figure 14. While not shown in Figure 14, one or more lens is generally 
interposed between the imaging plane and the conical mirror such that only those rays 

10 which are peirt of the desired parallel sheet or slice of rays are projected onto an 
imaging sensor or the imaging plane. 

It will be appreciated that the system can be adapted such that parallel rays 
from other directions can be directed to the image plane. In such systems, the sheet 
of parallel rays corresponds to a conical sheet around the axis z. Again, the rays are 

15 substantially parallel in a second direction which corresponds to the thickness of the 
sheet (At). 

A slice camera in accordance with the present invention is one which receives 
a thin sheet of parallel rays over a large field of view around an axis (e.g., 1 80 or 360 
degrees) and projects these rays onto a large number of image pixels over an area of 

20 an image sensor with a respective angular displacement among at least a majority of 
the pixels. This is illustrated in Figure 1 5, wherein the acquired image points from a 
portion of a 360 degree field of view slice are mapped over the angle of rotation from 
61 and 62 into the wide disc-shaped section of the rectangular imaging sensor defined 
by the region R1-R2. This mapping of the thin sheet of parallel rays onto the pixels 

25 (1502-1528) in a large portion of the sensor area and distributed over the rotational 
angle 6 provides enhanced resolution for the present slice cameras. This can be 
represented by intensity values in the graph of Figure 16B. Iii contrast, if a 
conventional vsdde angle imaging system were used to capture an equivalent width 
section of the scene, that portion of the image would map to only to a single circle. 



(5) 




5 



where k is a constant of integration. For the case of Z=0 at r=0, k=c/2 which 
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e.g., that defined by Rl, and would be sampled by far fewer pixels on the imaging 
sensor, as can be seen in the intensity graph of Figure 16A. 

Referring to Figure 16B, it can be appreciated that the pixels distributed 
within the region 61, 62 and the width R1-R2 exhibit a non-uniform distribution over 
5 6. By interpolating these data points and then re-sampling the interpolated data, a 
uiiiform distribution of interpolated samples can be provided, if desired. 

Also, referring to Figure 15, it is apparent that the rays from the image 
slice are directed onto only a known portion of the imaging sensor (in the case of 
Figure 15, an annular ring). This can be used advantageously to form lower cost or 

10 higher resolution imaging sensors for a slice camera. To form a lower cost sensor, 

the pixels outside of the active portion of the sensor need not be provided. To form a 
higher resolution sensor, the same number of pixels used in a conventional 
rectangular sensor can be concentrated within the active portion of the sensor. Of 
course, conventional rectangular sensors can also be used in the practice of the 

1 5 present invention. 

Figure 17 is a schematic diagram illustrating a first embodiment of a slice 
camera. The slice camera employs a conical mirror section 1702 which receives 
parallel rays over a small slice width. In this case, the system is intended to be 
orthographic such that At = Az, where 2 is the vertical axis of the conical mirror, 

20 which is normal to the imaging plane. As discussed in connection with Figure 14, the 
conical mirror orthogonally directs the incident rays onto a relay lens 1704. The 
relay lens 1704 directs the rays onto the entrance pupil of an imaging lens 1706 which 
in turn directs the rays onto an imaging sensor 1708. This arrangement of the relay 
lens and imaging lens can be referred to as a telecentric lens. In this arrangement, the 

25 parallel rays (and only the parallel rays) from the width of the slice are directed onto 
the image sensor. The parallel rays are directed onto an area of the imaging sensor 
1708 in an annular ring having a radial thickness as illustrated in Figure 15. Since the 
relay lens ensures that only orthographically reflected rays are imaged by the sensor, 
and since the mirror is a cone with its axis aligned with the direction of orthographic 

30 reflection, the scene rays that are imaged in any given angle within the 360 degree 

strip are all parallel to one another. If the conical section is small, these parallel rays 
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will be very close to one another. 

A variety of other slice camera designs may be used to capture high resolution 
slice data. For instance, instead of a conical mirror and an orthographic imaging 
system, a perspective imaging system can be used with a non-convex mirror as shown 
in Figure 18. As developed in connection with Figure 13, such an embodiment would 
include a section of mirror 1802 with a profile suitable for focusing the incident 
parallel rays onto the entrance pupil of an objective lens 1804 and ultimately onto the 
imaging sensor 1806. Further, the same data can be obtained using a stacked set of 
thin mirror sections. 1902, 1904, 1906 that have the same field of view but are of 
different diameters, as illustrated in connection with Figure 19. For example, the 
mirror sections can take the form of a stack of parabolic mirrors used in conjunction 
with an orthographic imaging system similar to that described in connection with 
Figure 17. 

Figure 20A illustrates a further alternate embodiment of a slice camera which 
employs a primary mirror 2002 and a secondary mirror 2004 to direct the incident 
parallel rays of the image slice onto the entrance pupil of an imaging lens 2012 and 
onto an image sensor 2006. As illustrated in Figure 20A, the primary mirror 2002 
can take the form of the hat shaped profile as discussed in connection with Figure 13 
and the secondary mirror can be an ellipsoid mirron In this case, the profile of the 
primary mirror is determined such the incident rays are directed through the near 
focus of the ellipsoid mirror and the entrance pupil of lens 2012 is placed at the far 
focus of the ellipsoid mirror 2004. This provides a configuration where the image 
sensor 2006 is placed within the perimeter of the primary mirror, thus providing a 
relatively compact design. 

Figure 20B illustrates another alternate embodiment of slice camera using 
multiple mirrors. In this case, the parallel rays of the desired image sheet are incident 
on a conical mirror section which directs the rays, which are still parallel, onto a 
second mirror 20 1 0 which is parabolic. The parabolic mirror 20 1 0 then focuses the 
parallel rays onto the entrance pupil of an imaging lens 2012 and onto an imaging 
sensor 2006. 

In the implementations discussed above, the fields of view (strips) or slice 
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widths are approximately perpendicular to the optical axis of the imaging system. It 
should be appreciated, however, that the field of view or slice width can be at any 
other angle with respect to the optical axis. For example, as noted in connection with 
Figure 12, this results in the image slice data corresponding to a conical sheet of 
5 thickness At around the axis of the imaging system (z). 

It wi 11 be appreciated that, as with conventional imaging systems, resolution 
can be improved by projecting a smaller field of view onto an equivalent area of the 
imaging sensor. For example, by configuring the optics such that the sheet of rays 
. over a 1 80 degree field of view are projected onto a substantial area of a rectangular 

10 imaging sensor, the resolution can effectively be increased as compared to a 360 
degree field of view camera. In such a case, only a half annular image is acquired 
and the camera is then rotated 360 degrees, rather than 1 80 degrees, in order to 
capture slice image data for a full 360 x 360 mosaic. In the case where the aspect 
ratio of the imaging sensor is not equal to one, such as in conventional video cameras, 

15 the sheet of rays can be made to fill the longer of the two image dimensions to more 
.. effectively use the area of the sensor and thereby obtain higher resolution (more 
pixels in the image section). Of course, a further extension of this principle is to limit 
the panorama or mosaic image to less than 360 x 360 viewing. For example, the 
magnification of the imaging lens can be increased such that rays from smaller 

20 sections of the mirror (e.g., 90 degrees) are directed to an equivalent area of the 
imaging sensor. 

A 1 80 degree or greater image slice can also be captured using a line image 
detector. Line detectors have a linear array of pixels closely packed together. Figure 
21 illustrates a slice acquisition system using a line detector 2102, an objective lens 
25 2 1 04 and two strips of curved mirrors 2 1 06, 2 1 08. Each scene ray that approaches 
the center of projection of the imaging system is reflected by the primary curved 
mirror 2108 in the direction of the secondary mirror 2106. The secondary mirror 
2106 then reflects this ray towards the entrance pupil of the imaging lens 2104. All 
reflected rays lie on a plane so that the resulting image is formed on the line sensor. 



30 



If the direction of the scene ray is s, the direction of the reflected ray v, the 
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direction of the line joining the reflecting points is m, the surface normals at the 
reflection points are ni and n2 are: 



s- m 



n 



(6) 



m 



v+ m 



(7) 



v+ m 



5 



Figure 22 is a schematic diagram of an exemplary system for generating and 



manipulating spherical mosaics in accordance with the present invention. The system 
includes a slice camera 2202 which is rotated about an axis of rotation 2204 by a 
motorized drive unit 2206. The slice camera 2202 can take any number of forms, 
including those illustrated in Figures 17-20. A control unit 2208 is provided to 

10 receive the slice image data from the slice camera 2202 and to control the rotation of 
the drive unit 2206. The drive unit 2206 can be a linear motor and include a sensor 
to provide rotational positional information to the control unit 2208. Alternatively, 
the drive unit 2206 can take the form of a stepper motor which induces a fixed 
amount of rotation in response to a received control signal firom the control unit 2208. 

15 In either case, the image acquisition by the slice camera is performed at known 

rotational positions about the axis of rotation 2204. As illustrated in Figure 22, the 
sKce camera 2202 can be offset firom the axis of rotation 2204 by a small radial 
distance, designated as b/2. As will be described below, this allows stereoscopic 
images to be rendered firom a single rotation about the axis of rotation 2204. 

20 The system fiirther includes a display unit 2210 and user input device 2212, 

such as a digital pointer or "mouse". The display unit 2210 is used to display the 
spherical mosaic images to a user and also operates in cooperation with the input 
device 2212 and control unit 2208 to form a graphical user interface to the system. 
The control unit 2208, user input device 2212 and display 2210 can be embodied in a 

25 conventional personal computer platform operating under the control of appropriate 
software. 



Methods for creating an omnidirectional mosaic/image using rotating 
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catadioptric imaging systems will now be described. Two types of imaging systems 
have been described above. The first system type is for the acquisition of image 
strips. In the case of image strips, it is possible to acquire overlapping strip image 
data without knowledge of the rotational position of the imaging system during image 
5 acquisition. Figure 23 is a simplified flow chart illustrating a method of generating a 
spherical mosaic using such overlapping, strip images. Referring io Figure TA^ as the 
strip image system is rotated about the axis of rotation (z-axis in Figure 7A), strip 
images are acquired at various rotational positions (<|>) about at least 1 80 degrees of 
rotation (step 2305). 

1 0 Next, the overlapping, strip image data is processed using conventional image 

processing techniques to identify common features in the image strips to determine 
the relative alignment of the image strips in the spherical mosaic (step 2310). A 
variety of methods can be used for feature identification and alignment. For example 
a sum^of-square-difference (SSD) measure can be used to identify similar windows 

15 within consecutive images. To determine the similarity measure of a feature window 
centered at a location (x,y) in a first image (k) and a window centered at a point (i j) 
in the second image (K-1), the SSD measure ( for a 5 pixel x 5 pixel window) is 
computed as: 

E (i,j)=i; E (8> 

m=-2n=-2 

20 where, Ij, and denote brightness/color values in the two images. The value of {ij) 
in image k-X that gives the lowest value of E is taken to be the best match for the 
feature (x, y) in image k. This process is repeated so that a significant number of 
features are matched. Note that each image feature can be mapped to a vector P in its 
reference frame. Therefore, the set of matching features correspond to a set of 

25 matching vectors in the frames k and * - 1 . Hence, equation (1 8) (infra) can be used 
to solve for the rotation matrix R^./^^ using a variety of techniques. The rotation 
matrix can either be solved using linear, nonlinear, or search techniques. In our case, 
we have used the SIMPLEX search algorithm to find the a^^^ Pa and values that best 
satisfy equation (18) (infra) for given set of matching features. Note that since the 
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rotation between all consecutive frames can be computed in this manner, the rotation 
Rj, of the most recent frsime with respect to the coordinate frame of the mosaic (first 
frame) is easily detemiined using equation (16) (infra). Once common features are 
identified, these features can be used as a datum for combining overlapping strips into 
5 the spherical mosaic (step 23 1 5). A blending algorithm, discussed above, can then be 
used to adjust the color and intensity of the spherical mosaic in those regions where 
adjacent strips overlap (step 2320). Finally, the resulting spherical mosaic data is 
mapped to a display (2325). 

In the case of unknown rotations, such as the method of Figure 23, it is 

10 expected that the estimated rotations will have errors due to inaccuracies in the 

feature matching and estimation procedure. As a result, the blending in the overlap 
regions will not be perfect and will tend to give rise to blurring of the mosaic. This is 
because the multiple measurements in the overlap regions will be displaced with 
respect to their desired locations and hence will not correspond to exactly the same 

15 scene point. To overcome this problem, a fine matching algorithm can be used. The 
fine matching algorithm involves searching through the rotation space in a small 
neighborhood around the estimated rotation parameters for the best match between 
the two strips in their overlap region. In other words, the strip k is recomputed for a 
finite number of samples within the rotation space: 

20 a - 6a < a' < a + 6a (9) 

P - 6P < P' < P + SP (10) 
Y - < Y' < Y 6y (lOA) 



For each rotation used, the overlapping regions of the two strips are compared using 
the sum-of-square-difFerence (SSD) given by equation (8). In this case, the SSD 
25 value E is computed using the entire region of overlap between the two strips (and 
not just a small window). The set of rotation angles d\ p' and y' that minimize E is 
taken to be the correct rotation estimate and the corresponding position of the strips is 
used to blend the strips together. Again, blending is performed using weights that 
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correspond to the distances and ^2 ^ mosaic pixel from the closest boundaries of 
the strips. 

In the case of image slices and non-overlapping image strips, the rotational 
position of the imaging sensor must be known for each acquired image slice. The 
5 system described in connection with Figure 22 is suitable for such an application. 
Figure 24 is a flow chart which illustrates the process of generating and displaying 
spherical mosaic data from non-overlapping image slice data. First, non-overlapping 
image slices are acquired at known rotational positions about the axis of rotation of 
the imaging system (step 2405). The rotational positions can be acquired using an 
10 ' appropriate sensor and/or through a drive mechanism which is calibrated with respect 
to an applied control signal, as discussed in connection with Figure 22. With the 
rotational position of the image slice data known, adjacent image sHce data is 
concatenated over the full rotational range of the imaging system (step 2410). As the 
r image slices are not overlapping, there is likely small regions between adjacent image 
15 . slices which are not captured. Therefore, interpolation can be performed to smooth • 
n the regions between image slices, if necessary (step 241 5). Finally, the spherical 
.:; mosaic data is then mapped to a display (step 2420). 

The image data acquired in step 2405, which is of known rotational position, 
must be mapped to the spherical mosaic. This entails mapping from locations in the 
20 image to corresponding locations within the reference frame of the mosaic, such as 
shown in Figure 1 A. In this reference frame a point may be represented by its polar 
angles (0y(p) or the Cartesian coordinates of the corresponding unit vector P [Xy 
Z]^, The specific mapping will depend on the catadioptric imaging system used. 
Figures 25A and 25B illustrate an exemplary mapping in the case of an imaging 
25 system which uses a parabolic mirror section, a relay lens, an objective lens and an 
image sensor. In this case, the projection of rays from the mirror to the image sensor 
is orthographic, i.e., all reflected rays are parallel to the optical axis as shown in the 
figure. With the Z-axis of the mosaic frame aligned with the optical axis of the 
imaging system, the azimuthal angle of the scene ray can be determined from the 
30 image coordinates (x, y): 



wo 01/22728 PCT/USOO/25801 

24 

^=tan"'— (11) 

X 

The distance of the reflecting mirror point from the optical axis is: 



r=-ylx^ + y^ (12) 



The profile of the parabolic mirror section is given by: 



5 ^^''>=""2>r" ^^^^ 

where, h is the parameter of the parabola. Hence, the zenith angle of the scene ray is 
determined as 

^=tan"^- (14) 
z 

Note that once the polar angles i0,<p) are determined the points can readily be 
1 0 mapped to the Cartesian coordinates P[Jl, Y,Z]^ using: 

X=sin^cos^ 
y= sin ^ sin ^ 
Z = cos^ 



As mentioned previously, the catadioptric strip or slice camera is rotated to 
capture a sequence of images. The frame of reference of the mosaic is generally 
defined with respect to the first orientation (or image) of the imaging system. Then, 
15 all subsequent measurements are mapped to this coordinate frame, taking the rotation 
into consideration. To do this, two coordinate frames are defined: the first being the 
fixed frame of the computed mosaic and the second being a fi-ame that this attached to 
the imaging system and is rotated with respect to the mosaic frame. We will denote 
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the rotation of orientation k with respect to the fixed frame as Rj^. This rotation can 
be computed as the product of the rotations between subsequent orientation Rk.i.i;: 

Each rotation between consecutive frames can be described in terms of the Euler 
5 angles (a^, Yk): 



1 0 



0 



0 cosa^ -sina^ 
0 sina^ cosa^ 



cosy^^ 0 sm fi^ 
0 1 0 

siny^^ 0 cos/?jt 



cosr* - sin;'^ 0 
0 0 1 



7) 



where, is a rotation about the Z-axis, is a rotation about the new 1^-axis, and Yh 
is a rotation about the new X-axis. Therefore, a point in the k!^ is easily mapped to 
the A: - 7'* frame by: 

p*-.=R*-,.*n (18) 



Once the rotation matrices are computed from the Euler angles, the image 
measurements can be mapped to the mosaic. The mosaic itself may be rendered on a 
sphere or a spherical panorama or any other representation of the polar angles {0,(p) 
as illustrated in Figure 1. 

Either image slices or strips (as defined earlier) produced by an imaging 
system can be mapped to constmct a complete mosaic from a single rotation of the 
imaging system. In the case of slices, which cannot overlap, the slices may be 
concatenated. In the case of image strips, the strips may either be concatenated or 
blended together where consecutive strips overlap. 

Once an omnidirectional mosaic has been computed, such as described in 
connection with Figures 23 and 24, any part of this mosaic can be mapped to a 
substantially distortion free perspective view of the corresponding part of the scene. 
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This can be done via a geometrical mapping method. Referring to Figure 26, a 
virtual camera view is defined by the polar angle & ^ <f>* of the viewing direction of 
the virtual camera. These angles can be mapped to the viewing unit vector P' using 
equation (15). For an effective focal length of the virtual camera off and the desired 
5 perspective having its two axes aligned with the unit vectors a and b which are 

orthogonal to each other and orthogonal to the viewing vector P', a pixel location (p, 
q) in this image is mapped to the mosaic frame of reference by computing the 
corresponding unit vector Q = P / II P II where: 

P=yp'+pa + qb (19) 
10 Once Q has been computed, the corresponding polar angles can be determined from 
equation (15). Once the polar angles are determined, the corresponding brightness 
value can be obtained from the representation used to store the mosaic. This 
brightness value is filled in the location (p, q) of the virtual camera image. 

The geometrical mapping can be performed in real time such that a user can 
1 5 vary the viewing vector and the focal length using an interactive device such as a 
computer mouse. The resulting virtual camera image can be displayed on the 
computer screen and the user can fi-eely navigate around the 360 degree x 360 degree 
space. Alternatively, a head-mounted display may be used to view the generated 
mosaic. As the user turns, the head mounted display provides the computer the 
20 orientation (viewing direction) of the user. The appropriate perspective image can 
then be computed and displayed on the head-mounted display. 
Stereoscopic Omnidirectional Mosaics 

The methods presented above can be used to develop an omnidirectional 
mosaic that is stereoscopic. This allows a user to "look around" the scene while 

25 perceiving the relative depths of objects in the scene. It is well known that 

stereoscopic viewing requires two different views of the scene to be captured so that 
the two different views can be presented to the two eyes of the observer to produce 
depth perception. As noted in connection with Figure 22, this principle can be used 
in the context of spherical mosaics to generate a stereoscopic representation of the 

30 scene that permits user to look in any direction. 
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Figures 27 A-C illustrate an imaging system being rotated about an axis that is 
radially offset with respect to the center of projection b, such as is illustrated in 
Figure 22. The radial distance between the rotation axis and the center of projection 
' is b/2. The acquired image sections can take the form of strips or slices. As is 
5 illustrated in Figure 27, the center of projection travels along a circle defined by 
radius b as the system is rotated through a range of 360 degrees. 

Figure 27A illustrates the left and right views, offset by 180 degrees of 
rotation, for each scene point that are produced by a single off-axis rotation of the 
imaging system. For the image that is captured at angle i|f and i|r + ic, the left section 

10 of field of view strip at angle i|; and the right section of strip at angle ijf + tc 

correspond to two different views of the same part of the scene captured with an 
approximate baseline of Z?. Thus, left and right strips (or slices) that are taken from 
images that are approximately diametrically opposite to one another have a baseline b 
:: and all scene points are viewed from two view points that have a baseline of b. As a 

15 ' result, two sphericat mosaics can be constructed from this single rotation. One 
mosaic can be constmcted from the sequence of 180 degree left strips (or slices) 
(denoted by L) and the other from the sequence of 1 80 degree right strips (or slices) 
(denoted by R). If the two mosaics are constructed simultaneously, the first strip in 
the left mosaic is positioned at 0= ijj and the first strip in the right mosaic is 

20 positioned at ^= ij; + tu. The image strips or slices are then combined using the 

mosaicing approaches discussed above, such as with reference to Figures 23 and 24. 

It should be noted that the field of view of the imaging system can be an 
image slice rather than a strip. In this case the left and right halves of the slice are 
concatenated together to construct the left and right mosaics, respectively. Also, as 

25 shovm in Figure 27B the angular displacement between the left and right views does 
not have to be 7C. For example, if the displacement is some arbitrary value Q, the two 
views are verged and have a baseline of 



In the embodiments discussed above, the image strip that is captured by the imaging 
30 camera is tangential to the circle of rotation. However, this does not have to be the 




(22) 
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case. The imaging system can be constructed such that the left and right strips (or 
slices) point away from the tangent to the circle of rotation as shown in Figure 27C. 

A property of the above method is that the stereoscopic information that is 
captured is complete (in field of view) and is accomplished with a single rotation of 
5 the imaging system. The two viewpoints for any scene point remain fixed on the 
circle of rotation. Moreover, the distance (baseline) between the viewpoints also 
remains fixed when the same parts of the image are used for the left and right views. 
Hence, the data that is obtained is sufficient to compute stereo views for the head 
movements shown in Figure 28A. All stereo data that results from panning and 

10 tilting the head is embedded in the spherical stereo mosaic. 

As illustrated in Figure 28B, there are two (left and right) mosaics that are 
used to compute the left eye and right eye perspective views for any given head 
orientation. These views can be displayed on the left and right displays of the head- 
mounted display so that the three-dimensional structure of the scene can be observed 

15 by a user. If a head-mounted display is not available for stereoscopic viewing of the 
mosaics, passive stereo glasses may be used where the left side of the glass has a red 
filter and the right side has a blue filter. Then, the perspective stereo image is 
displayed on the computer screen by taking the red channels of the perspective view 
from the left eye mosaic and the green and blue channel from the right eye mosaic. 

20 When navigating around the stereo mosaic, the user will perceive tiuree-dimensional 
structure while freely exploring the scene. 

It should be noted that a catadioptric imaging system is not required for the 
capture of spherical stereo mosaics. A very wide angle imaging system, such as one 
that uses a fish eye lens, can also be used to capture the images. Again, in this case, 

25 there exists a unique mapping from measured image points (x, y) to the polar angles 
i0y<p) that are defined in the cartesian reference frame P[X, y, ZJ^that is attached to 
the center of projection of the imaging system. Hence, 180 degree of 360 degree 
slices or strips of the field of view can be obtained from this image data and once 
these slices and strips are mapped to the mosaic representation, they can be processed 

30 in the same manner as described for the catadioptric imaging system. For a simple 

spherical mosaic, the fish eye system is rotated about its center of projection and for a 
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stereo mosaic it is rotated about a position which is radially offset from its center of 
projection. Again, as in the catadioptric case, the rotations may be known or 
unknown and estimated using matching features in consecutive image frames. 
Omnidirectional Depth Maps 
5 The systems and methods described above for generating a stereoscopic image 

ran be used to generate omnidirectional depth maps. The imaging system used 
provides slices or strips that are at least 1 80 degrees in one of the two dimensions of 
the strip or slice. The easiest way to compute a depth map is to use the left and right 
stereo mosaics, which are shown as spherical panoramas in Figures 29 A and 29B. 

10 Note that the point qj and in the left and right mosaics of Figures 29a and 29B, 

respectively, correspond to the same scene point. A feature matching algorithm, like 
the one used for rotation estimation discussed in connection with Figure 23, can be 
applied to the two mosaics. This yields a matching feature in the right image for each 
::. feature in the left image. These matching features correspond to the same physical 

I S point in the scene, the three-dimensional coordinates of which can then be computed 
using the geometric model which is illustrated in Figure 29C. The two mosaic points 
are mapped to their corresponding polar angles which are in turn mapped to unit 
viewing vectors in Cartesian coordinates. The coordinate frames for the two vectors 
are different as the left and right images of this scene point were captured from 

20 different viewpoints resulting from the radial offset in rotation, (b/2). Hence, the two 
reference frames are separated by the bsiseline b \ The intersection of the lines / and r 
in the directions of the viewing vectors P, and P, determines the coordinates S of the 
scene point in three-dimensional space. By repeating this process for all features in 
the left and right mosaics, a complete depth map of the scene can be determined. 

25 As an alternative to using the left and right mosaics for the computation of 

depth, greater accuracy can be obtained by using the sequence of original image 
sections captured during rotation. These image sections are illustrated in Figure 30. 
Note that these images correspond to discrete orientations (denoted by • -^n in 

Figure 30) of the imaging system. If the imaging system produces a strip rather than 

30 a slice, the same scene point will be seen on one side of the strip (say left side) in a 

part of the sequence of images, then it will not appear for a part of sequence, and then 
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. will reappear on the other side of the strip for another part of the sequence. Note that 
all images of the point correspond to different views of the point. The corresponding 
viewing vectors are shown in Figure 3 1 . 

Referring to Figure 31, the bundle of lines ideally intersect at the same point 
5 S, which is the three-dimensional coordinates of the scene point. However, due to 
inaccuracies in the measurements as well as computations, it is likely that the lines 
will not meet at a point. This is almost always the case in any type of stereo 
measurement. Hence, the bundle of lines can be used to determined to scene point 
location that has the shortest average distance from the bundle of lines. The 

10 coordinates of the point S can be accurately computed from the bundle of lines using 
a standard regression method. Since different pairs of lines correspond to different 
baselines in this case, this is really a form of multi-baseline stereo. The computed 
three-dimensional map of the scene can be stored as a 360 degree x 36& degree 
spherical mosaic, as a spherical panorama, or any other convenient representation. 

15 An exemplary imaging system, substantially in accordance with Figure 22, 

has been implemented using a conical mirror having a 65 nun outer diameter in 
conjunction with a telecentric lens to direct image slices to a conventional digital 
video camera (Canon model Optura). The rotation system includes a motorized 
turntable manufactured by Daedal. The center of projection of the slice camera was 

20 offset from the center of rotation of the turntable by three inches, thus providing a 

stereo baseline separation of six inches. A complete rotation of the slice camera was 
performed over a three minute interval over which approximately 5,000 image slices 
were acquired. An example of an image slice acquired by the system is illustrated in 
Figure 32. Exemplary left and right spherical mosaic images generated from the 

25 image slices are illustrated in Figures 33 A and 33B respectively. Each mosaic is 4000 
X 2000 pixels. Figures 34 A and 34 B illustrate left and right perspective views, 
respectively, of a section of the spherical image. 

The systems and methods of the present invention have been described in 
connection with certain embodiments thereof. It will be appreciated that changes and 

30 modifications to these embodiments will be apparent to those skilled in the art. Such 
changes and modifications are encompassed by the present invention as set forth in 
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the appended claims. 
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CLAIMS 

1. A method of acquiring slice image data comprising: 

receiving rays over a large field of view in a first direction and a small 
imaging width in a second direction, at least a portion of said rays being parallel with 
5 each other with respect to said second direction; and 

directing only said portion of said rays upon an area of an imaging sensor. 

2. The method of acquiring slice image data according to claim 1, wherein said 
directing step includes interposing at least one mirror and at least one lens between a 
point where the rays are received and the imaging sensor. 

10 3. The method of acquiring slice image data according to claim 1, wherein said 
directing step distributes said portion of said rays upon the imaging sensor at points 
corresponding to a plurality of different radial directions of the large field of view. 

4. The method of acquiring slice image data according to claim 3, further 
comprising the step of interpolating between said points corresponding to a plurality 

15 of different radial directions. 

5. The method of acquiring slice image data according to claim 4, further 
comprising sampling the interpolated data over a uniform distribution of said radial 
direction. 

6. The method of acquiring slice image data according to claim 1, wherein the 
20 large field of view is approximately 360 degrees. 

7. The method of acquiring slice image data according to claim 6, wherein the 
area of the imaging sensor is an annular ring having a radial width. 



wo 01/22728 PCT/USOO/25801 

33 

8. The method of acquiring slice image data according to claim 7, wherein said 
directing step distributes said portion of said rays throughout the radial width at 
points corresponding to a plurality of different radial directions of the large field of 
view. 

5 9, The method of acquiring slice image data according lo claim. 1 , wherein the 
large field of view is approximately 1 80 degrees. 

10. The method of acquiring slice image data according to claim 9, wherein the 
area of the imaging sensor is a semi-circular annular portion having a radial width. 

11. The method of acquiring slice image data according to claim 10, wherein said 
10 directing step distributes said portion of said rays throughout the radial width at 

points corresponding to a plurality of different radial directions of the large field of 
view. 

12. A method of generating an omnidirectional mosaic image comprising: 

acquiring image slice data at a plurality of known rotational 
15 positions about an axis of rotation, the image slice data corresponding to an image 
section having a large field of view along the axis of rotation and a small image 
width in a second direction, the acquiring step further comprising receiving rays over 
the large field of view, at least a portion of which are substantially parallel in said 
second direction, and directing only said portion of rays onto at least a portion of an 
20 image sensor area; and 

concatenating the plurality of image slice data using the known 
rotational positions to form a substantially continuous mosaic image data about the 
axis of rotation. 

13. The method of generating an omnidirectional mosaic image of claim 12, 
25 further comprising the step of mapping the omnidirectional image to a display. 
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14. The method of generating an omnidirectional mosaic image of claim 12, 
wherein the large field of view is substantially equal to 360 degrees and wherein the 
parallel rays are directed on the image sensor in an annular ring pattern having a 
radial width. 

5 15. The method of generating an omnidirectional mosaic image of claim 14, 

wherein said directing step distributes said portion of said rays throughout the radial 
width at points corresponding to a plurality of different radial directions of the large 
field of view. 

16. The method of generating an omnidirectional mosaic image of claim 12, 

10 wherein the second direction is orthogonal to the axis of rotation and the parallel rays 
correspond to a planar sheet. 

17. The method of generating an omnidirectional mosaic image of claim 12, 
wherein the second direction is non-orthogonal to the first direction and the parallel 
rays correspond to a conical sheet. 

15 18. The method of generating an omnidirectional mosaic image of claim 1 2, 

wherein said directing step is performed using a conical mirror and telecentric lens 
arrangement. 

19. The method of generating an omnidirectional mosaic image of claim 12, 
wherein said directing step is performed using a non-convex mirror in cooperation 

20 with an imaging lens. 

20. The method of generating an omnidirectional mosaic image of claim 12, 
further comprising the step of interpolating between images following the step of 
concatenation. 



25 



21. A method of generating an omnidirectional mosaic image comprising: 



wo 01/22728 PCT/USOO/25801 

35 

acquiring a plurality of overlapping image strips about an axis of 
rotation, the image strips having a large field of view along the axis of rotation and a 
small field of view in a second direction; 

identifying common features in overlapping regions of consecutive 

5 image strips; 

combining consecutive image strips based on the common features 
to form a substantially continuous omnidirectional image about the axis of rotation; 
and 

blending the combined images in the overlapping regions to 
10 provide omnidirectional image data. 

22. The method of generating an omnidirectional mosaic image of claim 21 
further comprising mapping the omnidirectional image data to a display. 

23. The method of generating an omnidirectional mosaic image of claim 21 
wherein said plurality of strips is at least 3 strips. 

15 

24. A slice camera comprising: 

an imaging sensor; 

at least one mirror receiving rays over a large field of view in a 
first direction and a small imaging width in a second direction, at least a portion of 
20 said rays being parallel with each other with respect to said second direction; and 

at least one lens, the at least one lens receiving the rays from the at 
least one mirror and directing only said portion of said rays to an area of the imaging 
sensor. 

25. The slice camera according to claim 24 wherein the at least one mirror is a 
25 conical mirror and said at least one lens is a telecentric lens arrangement. 

26. The slice camera according to claim 24 wherein the at least one mirror is a 
non-convex mirror and said at least one lens is an imaging lens having an entrance 
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pupil, said non-convex mirror having a profile such that only said parallel rays are 
directed through the entrance pupil of the imaging lens and onto the imaging sensor. 

27. The slice camera according to claim 24 wherein the at least one mirror 
includes first and second mirrors. 

5 28. The slice camera according to claim 27 wherein the second mirror has a near 
focus and a far focus and wherein the first mirror section projects said portion of rays 
through the near focus and the second mirror directs said portion of rays to the image 
sensor located at a position proximate the far focus. 



29. The slice camera according to claim 28 wherein the near focus is located in 
1 0 front of the first mirror section and the far focus is located behind the first mirror 

section. 

30. The slice camera according to claim 28 wherein the second mirror is an 
ellipsoid mirror. 

31 . The slice camem according to claim 28 wherein the first mirror is a conical 
1 S mirror and said second mirror is a parabolic mirror. 

32. The slice camera according to claim 3 1 wherein the imaging sensor is located 
behind an apex of the conical mirror. 

33. The slice camera according to claim 24, wherein said portion of said rays are 
directed upon the imaging sensor at points corresponding to a plurality of different 

20 radial directions of the large field of view. 



34. The slice camera according to claim 24, wherein the large field of view is 
substantially equal to 360 degrees and wherein the area of the imaging sensor is an 
annular ring having a radial width. 
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35. The slice camera according to claim 34, wherein said rays are directed 
throughout the radial width at points corresponding to a plurality of different radial 
directions of the large field of view. 

36. The slice camera according to claim 24, wherein the large field of view is a 

5 sector less than 360 degrees and wherein the area of the imaging sensor is a section of 
an annular ring, corresponding to said sector, and having a radial Mddth. 

37. The slice camera according to claim 36, wherein said rays are directed 
throughout the radial width at points corresponding to a plurality of different radial 
directions of the large field of view. 

10 38. An imaging system for acquiring an omnidirectional mosaic image 
comprising: 

an image slice camera having a large field of view in a first 
direction and a small imaging width in a second direction, the image slice camera 
receiving rays of light incident on the camera vsdthin the large field of view, at least a 
15 portion of said incident rays being substantially parallel to each other v^th respect to 
said second direction, said portion of rays being directed to an area of an imaging 
sensor to form image slice data; 

a rotational drive system operatively coupled to the image slice 
camera, the rotational drive system rotating the image slice camera along an axis of 
20 rotation substantially parallel to the first direction; 

a control unit operatively coupled to the rotational drive system and 
image slice camera, the control unit receiving the image slice data from the image 
slice camera and recording corresponding rotational positional information of the 
rotational drive system, the control unit acquiring a plurality of image slices at known 
25 rotational positions and concatenating said image slices to form an omnidirectional 
mosaic image data. 



39. The imaging system of claim 38, further comprising an output device coupled 
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to the control unit for displaying an omnidirectional mosaic image from the 
omnidirectional mosaic image data. 

40. The imaging system of claim 38, wherein the slice camera comprises: 

an imaging lens having an entrance pupil; 
5 a non-convex mirror section receiving the parallel rays and 

directing said portion of rays onto the entrance pupil of the imaging lens; 

an imaging sensor, the imaging sensor being responsive to the rays 
which are incident on the entrance pupil of the imaging lens. 

41 . The imaging system of claim 38, wherein the slice camera comprises: 
10 an imaging sensor; 

a telecentric lens arrangement; and 

a conical mirror section, the conical mirror section receiving said 
incident rays and directing said portion of rays onto the telecentric lens arrangement 
whereby only said portion of rays are directed to the imaging sensor. 

15 42. The imaging system of claim 38, wherein the slice camera comprises: 

a first mirror section receiving said incident rays; 
a second mirror section; and 

an imaging sensor, the second mirror being positioned to receive 
said portion of rays fi-om the first mirror and direct said portion of rays to the imaging 
20 sensor. 

43. The imaging system of claim 42, wherein the second mirror section has a near 
focus and a far focus and wherein the first mirror section projects said portion of rays 
through the near focus and the second mirror section directs said portion of rays to the 
image sensor located at a position proximate the far focus. 



25 



44. The imaging system of claim 43 wherein the near focus is located in front of 
the first mirror section and the far focus is located behind the first mirror section. 
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45. The imaging system of claim 38, wherein the slice camera comprises: 

an imaging sensor; 

a telecentric lens arrangement; and 

a plurality of stacked parabolic mirror sections receiving the 
5 incident rays and directing said portion of rays to the telecentric lens arrangement 
such that only said portion of rays are incident on the imaging sensor. 

46. The imaging system of claim 38, wherein the slice camera has a center of 
projection and wherein the axis of rotation is radially offset from the center of 

10 projection. 

47. The imaging system of claim 38, wherein the control unit generates left and 
right omnidirectional mosaic data from the image slices acquired in a single rotation 
about the axis of rotation. 



48. The imaging system of claim 47, wherein said control unit provides the left 
1 5 and right omnidirectional mosaic data to the output device to form a stereoscopic 

image. 

49. The imaging system of claim 48, wherein the output device includes left and 
right display portions for displaying the left and right omnidirectional mosaic data, 
respectively. 

20 50. The imaging system of claim 49, wherein the output device is a head mounted 
display. 



5 1 . The imaging system of claim 46, wherein control unit calculates the depth of a 
plurality of points in the image from the left and right omnidirectional mosaic data. 
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Bureau or which the applicant did not request the receiving Office to prepare and transmitto the Ini^ 

as provided by Rule 17.1 (a) or (b), respectively. In such a case, the attention of the applicant is directed to Rule 17.1 (c) which 
* provides that no designated Office may disregard the priority claim concerned before giving the applicant an opportunity, 
upon entry into the national phase, to furnish the priority document within a time limit which is reasonable under the 
circumstances. 



Prioritv date 

20 Sept 1999 (20.09.99) 
13 Dece 1999 (13.12.99) 



Priority ap plication No. 

60/154722 
60/170,392 



Countrv o r maional Office 
or PCT re ceiving Office 

US 

us 



Date of receipt 
nf prinritv document 

22 Janu 2001 (22.01.01) 
22Janu2001 (22.01.01) 



*lTie International Bureau of WlPO 
34, chemin des Colombettes 
1211 Geneva 20, SwHzeriand 



Facsimile No (4V22) 740.14.35 



Authorized officer 

V. Gross 

Telephone No. (41-22) 338.83.38 



Form PCT/IB/304 (July 1998) 




003801934 



^HIS ^GE BUNK (uspto) 



jUi^ent cooperation trea 



From the 

INTERNATIONAL PREUMINARY EXAMINING AUTHORITY 



To: 



HENRY TANG 

BAKER BOTTS LLP 

30 ROCKEFELLER PLAZA 

NEW YORK NY 10112-0228 



per 



BAKF? POTTS L.L.R 
01 HAY \li pn 2' 21 



NOTIFICATION OF RECEIPT y q 
OF DEMAND BY COMPETENTj-tNTERi*ATI0N 
PRELIMINARY EXAMINING-AUTHQRliy': 

^ i I 

(PCT Rule 593(e) and 61.1(b). first sentence 
and Administrative Instructions, Section 601(a)) 






08 MAY 2001 ^ 


Applicant's or agent's file reference 

32709 -PCT 


IMPORTANT NOTIFICATION 


International application No. 

PCT/USOO/25801 


International filing date (dayAnonth/year) 

20 SEP 00 


Priority date (day/month/year) 

20 SEP 99 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF 
NEW YORK 



1, The applicant is hereby notified that this International Preliminary Examining Authority considers the following date as the 
date of receipt of the demand for international preliminary examination of the international application: 



2. That date of receipt is: 

the actual date of receipt of the demand by this Authority (Rule 61.1(b)). 

actual date of receipt of the demand on behalf of this Authority (Rule 59.3(e)), 

the date on which this Authority has, in response to the invitation to correct defects in the demand (Form 
PCT/IPE A/404), received the required corrections. 



aaie oi r« 



□ 



ATTENTION- That date of receipt is AFTER the expiration of 19 months from the priority date. Consequently, the 
election{s) made in the demand does (do) not have the effect of postponing the entry into the national phase until 
30 months from the priority date (or later in some Offices) (Article 39(1)). Therefore, the ^"fiy^;^^ 
national phase must be performed within 20 months from the priority date (or later m some Offices) (Article 22). 
For details, see the PCT Applicant's Guide, Volume 11. 

(If applicable) This notification confirms the information given by telephone, facsimile transmission or in person 
on: 



□ 



4. Only where paragraph 3 applies, a copy of this notification has been sent to the Internaiional ^"""^Jf;^ '"'OQCSJ fO' 




Name and mailing address of the IPEA/US 
Assistant Commissioner for Patents 
Box PCT 



Washington, D.C. 20231 
Facsimile No. 



Atm: IPEA/US 



Authorized officer 

Telephone ^^^gg^^g^/^^T^/ 



Form PCT/IPEA/402 (July 1998) 
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ENT COOPERATION TREA 



PCT/USOO/25801 



From the INTERNATIONAL BUREAU 



PCT 

NOTIFICATION OF THE RECORDING 

(PCT Rule 92bis.1 and 
MuiTiinisiroii V6 insxruciions, oeciion *^4L4L} 


To: 

TANG, Henry 
Baker Botts LLP 
30 Rockefeller Plaza 
New York, NY 10112-0228 
ETATS-UNIS D'AMERIQUE 


Date of mailing (day/month/year) 
16 May 2001 (16.05.01) 


Applicant's or agent's file reference 
32709-PCT 


01HAY3U A«iu:ui* 

IMPORTANT NOTIFICATION 


International application No. 
PCT/USOO/25801 


International filing date (day/month/yeaT)~~|'^'7]^^T]^g^^ 

20 September 2000 {20.b9.«e)^^^^i^^^^ 



1. The following indications appeared on record concerning: 

X . the applicant X the inventor [ | the agent | | the common representative 


Name and Address 

KARMAKAR, Amruta 
Department of Computer Science 
Columbia University 
450 Mudd Hall 
New York, NY 10027 
United States of America 


State of Nationality 
** 


State of Residence 
US 


Telephone No. 


Facsimile No. 


Teleprinter No. 


2- The International Bureau hereby notifies the applicant that the following change has been recorded concerning: 
1 1 the person X the name | X| the address X the nationality | | the residence 


Name and Address 

KARMARKAR, Amruta 
1916 Cambridge Court, Apt. 2B 
Palatine, IL 60074 
United States of America 


State of Nationality 
IN 


State of Residence 

us 


Telephone No. 


Facsimile No. 


Teleprinter No. 


3. Further observations, if necessary: 


A. A copy of this notification has been sent to: 
1 X[ the receiving Office [ X| the designated Offices concerned 
1 1 the International Searching Authority | | the elected Offices concerned 
I I the International Preliminary Examining Authority | | other: 



Authorized officer 

Marie-Th6r6se Priser 
Telephone No.: (41-22) 338.83.38 



The International Bureau of WIPO 
34, chemin des Colombettes 
1211 Geneva 20« Switzerland 



Facsimile No.: (41-22) 740.14.35 



Form PCT/IB/306 (March 1994) 



004026892 
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PCT/USOO/25801 



From the INTERNATIONAL BUREAU 



PCT 



INFORMATION CONCERNING ELECTED 
OFFICES NOTIFIED OF THEIR ELECTION 



(PCT Rule 61.3) 



Date of mailing (day/month/year) 



To: 






TANG, Henry 






Baker Botts LLP 






30 Rockefeller Plaza 






New York, NY 1 01 1 2-0228 






ETATS-UNIS D'AMERIQUE 













Applicant's or agenf s file reference 

32709-PCT 


IMPORTANT INFORMATION^^i^AS/ 


International application No. 
PCT/USOO/25801 


International filing date (day/month/year) 
20 September 2000 (20.09.00) 


Priority date (day/month/year) 

20 September 1999 (20.09.99) 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK et al 



1. The applicant is hereby informed that the International Bureau has, according to Article 31 (7), notified each of the following 
Offices of its election: 

EP :AT,BE,CH,CY,DE,DK,ES,FI,FR,GB,GRJEJT,LaMC,NL,PT,SE 

National :AU,BG,CA,CN,CZ,DEJLJP,KP,KR,MN,NO,NZ,PL,RO,RU,SE,SK,US 

2. The following Offices have waived the requirement for the notification of their election; the notification will be sent to them 
by the International Bureau only upon their request: 

AP :GH,GM,KE,LS,MW,MZ,SD,SL,SZ,TZ,UG,2W 
EA :AM,AZ,BY,KG,KZ,MD,RU,TJ,TM 

OA:BF,BJrCF,CG,CI,CM,GA,GN,GW,ML,MR,NE,SN,TD,TG 

National :AE,AG,AUAM,AT,AZ,BA,BB,BR,BY3Z,CH,CR,CU,DK,DM,DZ,EE,ES,FI,GB, 

GaGE,GH,GM,HR,HU,IDJNJS,KE,KG,KZ,LC^K,LR,LS,LT,LU^V,MA,MD,MG,MK,MW, 

MX,MZ,PT,SD,SG,SI,SL,TJ,TM,TR,TT,TZ,UA,UG,UZ,VN,YU,ZA,ZW 

3. The applicant is reminded that he must enter the "national phase" before the expiration of 30 months from the Pnority date 
before each of the Offices listed above. This must be done by paying the national fee(s) and furnishing , if prescribed, a 
translation of the international application (Article 39(1 )(a)), as well as, where applicable, by furnishing a translation of any 
annexes of the international preliminary examination report (Article 36(3)(b) and Rule 74.1). 

Some offices have fixed time limits expiring later than the above-mentioned time limit. For detailed information about the 
applicable time limits and the acts to be performed upon entry into the national phase before a particular Office, see Volume ii 
of the PCT Applicant's Guide. 

The entry into the European regional phase is postponed until 31 months from the priority date for all States designated for 
the purposes of obtaining a European patent. 



COPY TO 



The International Bureau of WlPO 
34« chemin des Colombettes 
1211 Geneva 20« Switzerland 


Authorized officer: >^ 

Cpudio Bortonj 


1 


Facsimile No. (41-22) 740.14.35 


Telephone No. (41 -22) 338.83.38 / 
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FROM: 



FOREIGN MAIL DESK' 



sIT COOPERATION TREm 



PLEASE BE ADVISED THAT THERE AR 
(.>^ PIECES OF MAIL. 



WO 01/22728 
PCT/USOO;258Q1 

From the INTERNATIONAL BUREAU 



NOTICE INFORMINU 1 Hb AKKLlCANT OF THE 
COMMUNICATION OF THE INTERNATIONAL 
APPLICATION TO THE DESIGNATED OFFICES 

(PCT Rule 47.1(c), first sentence) 


'"tang. Henry OAKER BQTTS L.L.P. 

New York, NY 1 01 1 2-0228 , , 
ETATS-UNIS D'AMERIQUB4tj^^^dfl|^^^ 


Date of mailing (day/month/year) 
29 March 2001 (29.03.01) 




Applicant's or agent* s file reference 
32709-PCT 


IMPORTANT NOTICE 


International application No. International filing date (day/month/year) Priority date (day/month/year) 

PCT/USOO/25801 20 September 2000 (20.09.00) 20 September 1999 (20.09.99) 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK et al 



i 

1. Notice IS hereby given that the International Bureau has connmunicated, as provided in Article 20, the international application 
to the follovy/ing designated Offices on the date indicated above as the date of mailing of this Notice: 

AU,KP,KR,US 

In accordance with Rule 47.1(c), third sentence, those Offices will accept the present Notice as conclusive evidence that 
the communication of the international application has duly taken place on the date of mailing indicated above and no copy 
of the international application is required to be furnished by the applicant to the designated Office(s). 

!. The following designated Offices have waived the requirement for such a communication at this time: 

AE,AG,AUAM,AP,AT,AZ3A,BB,BG,BR,BY,BZ,CA,CH,CN,CR,CU,CZ,DE,DK,DM,DZ,EA,EE,EP,E 

FI,GB,GD,GE,GH,GM,HR,HUJD,ILJNJSJP,KE,KG,KZ,LC,LK,LR,LS,LT,LU,LV,MA,MD,MG,MK^ 

MN,MW,MX,MZ,NO,NZ,OA,PUPT,RO,RU,SD,SE,SG,SI,SICSLJJ,TM,TR,TT^^^ 
The communication will be made to those Offices only upon their request. Furthermore, those Offices do not require the 
applicant to furnish a copy of the international application (Rule 49.1 (a-bis)). 

\. Enclosed with this Notice is a cop^fthe international application as published by the International Bureau on 
29 March 2C01 (29.03.01) under No. WO 01/22728 

REMINDER REGARDING CHAPTER 11 (Article 31(2)(a) and Rule 54.2) 

If the applicant wishes to postpone entry into the national phase until 30 months (or later in some Offices) from the priority 
date, a demand for international preliminary examination must be filed with the competent International Preliminary 
Examining Authority before the expiration of 19 months from the priority date. 

It is the applicant's sole responsibility to monitor the 1 9-month time limit 

Note that only an applicant who is a national or resident of a PCT Contracting State which is bound by Chapter II has the 
right to file a demand for international preliminary examination. 

REMINDER REGARDING ENTRY INTO THE NATIONAL PHASE (Article 22 or 39(1)) 

If the applicant wishes to proceed with the international application in the national phase, he must within 20 months 
or 30 months, or later in some Offices, perform the acts referred to therein before each designated or elected Office. 

For further important information on the time limits and acts to be performed for entering the national phase, see the 
Annex to Form PCT/IB/301 (Notification of Receipt of Record Copy) and Volume II of the PCT Applicant's Guide. 





The International Bureau of WlPO 
34, chemin des Colombettes 
1211 Geneva 20, Switzerland 

Facsimile No. (41-22) 740.14.35 


Authorized officer P'/T'f 
Telephone No. (41-22) 338.83.38 



Form PCT/IB/308 (J>ily 1 996) 



3914452 
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>ENT COORERATION TRE W 

From the INTERNATIONAL%^^MjBQTf?<CC>- 



PCT 



NOTIFICATION OF RECEIPT OF 
RECORD COPY 

(PCT Rule 24.2(a)) 



To: 



01 JAN -2 AMIi= 55 



TANG, Henry 
Baker Botts LLP 
30 Rockefeller Plaza 
New York, NY 10112-0228 
ETATS-UNIS D'AMERIQUE 




Date of mailing (day/month/year) 

20 December 2000 (20.12.00) 



IMPORTANT NOTIFICATION 



Applicant's or agent's file reference 
32709-PCT, 



International application No. 

PCT/USOO/25801 



The applicant is hereby notified that the International Bureau has received the record copy of the international application as 
detailed below. 

Name(s) of the applicant(s) and State(s) for which they are applicants: 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK (for all designated 

States except US) 
NAYAR, Shree, K. et al (for US) 
International filing date 20 September 2000 (20.09.00) 

Priority date(s) claimed 20 September 1 999 (20.09.99) 

13 December 1999 (13.12.99) 



Sste of receipt or the recoru copy 
by the International Bureau 

List of designated Offices 



12 December 2000 (12.12.00) 



AP :GH,GM,KE,LS,MW,MZ,SD,SL,SZ,TZ,UG,ZW 
EA :AM,AZ,BY,KG,KZ,MD,RU,TJ,TM 

EP :AT,BE,CH,CY,DE,DK,ES,FI,FR,GB,GR,IE,IT.LU,MC,NL,PT,SE 
OA :BF,BJ,CF,CG,CI,CM,GA,GN,GW,ML,MR,NE,SN,TD,TG 

National :AE,AG,AL,AM,AT,AU,AZ,BA,BB,BG,BR,BY,BZ,CA,CH,CN,CR,CU,CZ,DE,DK,DM,DZ,EE, 
ES,FI,GB,GD,GE,GH,GM,HR,HU,ID,IL,IN,IS,JP,KE,KG,KP,KR,KZ,LC,LK,LR,LS,LT,LU,LV,MA, 
MD,MG,MK,MN,MW,MX,MZ,NO,NZ,PL,PT,RO,RU,SD,SE,SG,SI,SK,SL,TJ,TM,TR,TT,TZ,UA,UG,US, 
UZ,VN,YU,ZA,ZW 



COPY TO 




The International Bureau of WlPO 


Authorized officer: jjCCLt f^A^^^^^^-/^ 


34, chemin des Colombettes 


Marie-Therese Priser 


1211 Geneva 20, Switzerland 




Facsimile No. (41-22)740.14.35 


Telephone No. (41-22)338.83.38 



Form PCT/IB/301 (July 1998) 



003732343 
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Continuation of FQrm PCT/IB/SOt 
NOTIFICATION OF RECEIPT OF RECORD COPY 



Date of mailing (day/month/year) 

20 December 2000 (20.12.00) 


IMPORTANT NOTIFICATION 


Applicant's or agent's file reference 
32709-PCT 


International application No. 
PCT/USOO/25801 



ATTENTION 

The applicant should carefully check the data appearing in this Notification. In case of any discrepancy between these data 
and the indications in the international application, the applicant should innmediately inform the International Bureau. 

In addition, the applicant's attention is drawn to the information contained in the Annex, relating to: 
I X I time limits for entry into the national phase 
I I confirmation of precautionary designations 
I X| requirements regarding priority documents 
A copy of this Notification is being sent to the receiving Office and to the International Searching Authority. 



Form PCT/IB/301 (continuation sheet) (July 1998) 



003732343 



THIS IFAGE BLANK (USPTO) 



ANNEX ^>RM PCT/IB/301 



'(i^^ational application No. 



W PCT/USOO/25801 



INFORMATION ON TIME LIMITS FOR ENTERING THE NATIONAL PHASE 



The applicant is reminded that the "national phase" nnust be entered before each of the designated Offices indicated in the 
Notification of Receipt of Record Copy (Form PCT/IB/301) by paying national fees and furnishing translations, as prescribed by 
the applicable national taws. 

The time limit for performing these procedural acts is 20 MONTHS from the priority date or, for those designated States 
which the applicant elects in a demand for international preliminary examination or in a later election, 30 MONTHS from the 
priority date, provided that the election is made before the expiration of 1 9 months from the priority date. Some designated (or 
elected) Offices have fixed time limits which expire even later than 20 or 30 months from the priority date. In other Offices an 
extension of time or grace period, in some cases upon payment of an additional fee, is available. 

In addition to these procedural acts, the applicant may also have to comply with other special requirements applicable in 
certain Offices. It is the applicant's responsibility to ensure that the necessary steps to enter the national phase are taken in a 
timely fashion. Most designated Offices do not issue reminders to applicants in connection with the entry into the national 
phase. 

For detailed information about the procedural acts to be performed to enter the national phase before each designated 
Office, the applicable time limits and possible extensions of time or grace periods, and any other requirements, see the relevant 
Chapters of Volume 11 of the PCT Applicant's Guide. Information about the requirements for filing a demand for international 
preliminary examination is set out in Chapter IX of Volume I of the PCT Applicant's Guide. 

GR and ES became bound by PCT Chapter II on 7 September 1996 and 6 September 1997, respectively, and may, therefore, 
be elected in a demand or a later election filed on or after 7 September 1996 and 6 September 1997, respectively, regardless of 
the filing date of the international application. (See second paragraph above.) 

Note that only an applicant who is a national or resident of a PCT Contracting State which is bound by Chapter It has 
the right to file a demand for international preliminary examination. 



This notification lists only specific designations made under Rule 4.9(a) in the request It is important to check that these 
designations are correct. Errors In designations csn be corrected where precautionary designatioiis have been made under 
Rule 4.9(b). The applicant is hereby reminded that any precautionary designations may be confirmed according to Rule 4.9(c) 
before the expiration of 15 months from the priority date. If it is not confirmed, it will automatically be regarded as withdrawn 
by the applicant. There will be no reminder and no invitation. Confirmation of a designation consists of the filing of a notice 
specifying the designated State concerned (with an indication of the kind of protection or treatment desired) and the payment 
of the designation and confirmation fees. Confirmation must reach the receiving Office within the 15-month time limit. 



For applicants who have not yet complied with the requirements regarding priority documents, the following is recalled. 

Where the priority of an earlier national, regional or international application is claimed, the applicant must submit a copy 
of the said earlier application, certified by the authority with which it was filed ("the priority document") to the receiving Office 
(which will transmit it to the International Bureau) or directly to the International Bureau, before the expiration of 16 months from 
the priority date, provided that any such priority document may still be submitted to the International Bureau before that date of 
international publication of the international application, in which case that document will be considered to have been received 
by the International Bureau on the last day of the 16-month time limit (Rule 17.1(a)). 

Where the priority document is issued by the receiving Office, the applicant may, instead of submitting the priority 
document request the receiving Office to prepare and transmit the priority document to the International Bureau. Such request 
must be made before the expiration of the 16-month time limit and may be subjected by the receiving Office to the payment 
of a fee (Rule 17.1(b)). 

If the priority document concerned is not submitted to the International Bureau or if the request to the receiving Office 
to prepare and transmit the priority document has not been made (and the corresponding fee, if any, paid) within the applicable 
time limit indicated under the preceding paragraphs, any designated State may disregard the priority claim, provided that no 
designated Office may disregard the priority claim concerned before giving the applicant an opportunity to furnish the priority 
document within a time limit which is reasonable under the circumstances. 

Where several priorities are claimed, the priority date to be considered for the purposes of computing the 16-month time 
limit is the filing date of the earliest application whose priority is claimed. 



CONFIRMATION OF PRECAUTIONARY DESIGNATIONS 



REQUIREMENTS REGARDING PRIORITY DOCUMENTS 



Form PCT/IB/301 (Annex) (July 1998) 



003732343 
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PATENT COOPERATION TREATY 

PCT , 

INTEHNATIOXAL PHELIMINAHV EXAMINATION HE1X)HT M ^ ^ ^ 

(PCT Article 36 and Rule 70) . . . 



Applicant's or agent's file reference 
3*2709-PCT 


crfcw cfTUTwirw A#-TTnM See Notification of Transmittal of International 
FOR FURTHER AClloiM p^^ij^ingry ExaiTiination Report (Form PCT/IPEA/416) 


International application No. 
PCT/US0()/l'580I 


International filing date (day/ month/year) Priority date (day/month/year) 
^20 SEPTEMBER i>000 SEPTEMBER 


International Patent Classification (IPC) or national classification and IPC 
IPC(7): HO+N 5/i2i.>5 and US CI.: 348AJ07 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK 



This intoruatioual preliminary exaiuiiiatiou roport has been prepare<l by this Intornational Pn^liininary 
Examining Authority and is t.ransmitte<l JaJJie applicant at-conling to Article 36. 

This REPORT consists of a total oP 




sheets. 



□ This re|K>rt is also accoiui>ame<l by ANNEXES, i.e., sheets of the description^ claims and/br draAvings which have 
been amenilwl and are the basis for this re|Kjrt. asuVor sheets containing rectifications made before this Aiitht)rity. 
70.16 and Section 607 oi;^ 

consist of a total of i ^ s 



(see Rule 
These annexes 



Administnitive Instnictioiis under the KTI"). 
sheets. 



3. This report contains indications relating to the folloM'ing items: 



I 




Basis of the rciJort 




II 


□ 


Priority 




III 


□ 


Non-estahlishmcut of report. M ith rc^ganl to novelty, inventive stej) or 


i nd u str i a 1 app 1 i ca b i 1 i t.y 


IV 


□ 


Lack of unity of invention 




V 


[1 


Reasoned statement under Article^ ;i5(Ji) >vith ivganl to novelty, inventive 


stei> or industrial appliciil)ility; 




citations and ex]>lanatioi]s sup)>orting such statement 




w 


□ 


Certain dociiiucnts cite<l 




Vll 


□ 


Certain defe<*tK in the international application 




WW 


□ 


Certain oWnatious on the international application 





Date of submission of the demand 
18 APRIL *20()l 


Date of completion of th 
05 APRIL i!00*J 


s report 


Name and mailing address of the IPEA/US 
Cbiniiiissionor of PnriMits and 'IV»ilriiiiirks 
Box PCT 

Wiushingtou. D.i: !J<)5i3l 
Facsimile No. (703) 305-3*230 


Authorized officer 

LUONC; NGUYEN 
Telephone No. (703) J 


08-J>^2!)7 ^ 





Form PCT/IPEA/ KW (cover sheet) (July 1!>J)«)* 



THIS PAGE BLANK (uspto) 



INTERNATIONAL PRELIMINARY EXAMINATION REPORT 



International application No. 
PCT/USOO/25801 



1. Basis of the report 



1 . With regard to the elements of the international application:* 
[3c] the international application as originally filed 





the description: 
pages 


1-31 






. as originally filed 




pages 


NONE 






, filed with the demand 






NONE 




, filed with the letter of 






pages 

the claims: 
pages 


32-39 






, as originally filed 




pages 


NONE 




, as amended (together with any statement) under Article 19 




pages 


NONE 






, filed with the demand 






NONE 


filed with the letter of 






pages 

the drawings: 
paces 


1-22 






, as originally filed 




pages 


NONE 






« filed with the demand 






NONE 




, filed with the letter of 




[U 


the sequence listing part of the description: 

n^a^c NONE 




. as originally filed 




paees 


NONE 






. filed with the demand 




pages 


NONE 




, filed with the letter of 





2. With regard to the language, all the elements maiked above were available or furnished to this Authority in the language in which 
the international application was filed, unless otherwise indicated under this item. 

These elements were available or furnished to this Authority in the following language which is: 

I I the language of a translation furnished for the purposes of international search (under Rule 23.1(b)). 
I I the language of publication of the international application (under Rule 48.3(b)). 

I I the language of the translation fiimished for the puqwses of international preliminaiy examination (under Rules 55^ and/ 
or 553). 

3. With regard to any nucleotide and/or amino acid sequence disclosed in the international application, the international 
preliminary examination was carried out on the basis of the sequence listing: 

□ contained in the international application in printed form. 

I [ filed together with the international application in computer readable form. 

I I furnished subsequently to this Authority in written form. 

I I furnished subsequently to this Authority in computer readable form. 

□ The statement that the subsequently furnished written sequence listing does not go beyond the disclosure in the 
international application as filed has been furnished. 

I I The staternent that the information recorded in computer readable fomi is identical to the v^^iten sequence listing has 
' — ' been furnished. 

The amendments have resulted in the cancellation of: 

S the description, pases NONE 

CEl the claims, Nos. NONE 

ric] the drawings, sheets/ftg NONE 



4.L2i 



5. Q This report has been drav^ as if (some oO the amendments had no; been made, since they have been considered to go 

beyond the disclosure as filed, as indicated in the Supplemental Box (Rule 70.2(c)).** 
* Replacement sheets which have been furnished to the receiving Office in response to an invitation under Article 24 are referred to 
in this report as ''originally filed" and are not annexed to this report since they do not contain amendments (Rules 70, J o 



report 
and 70 J 7). 

**Anv replacement sheet containing such amendments must be referred to under item 1 and annexed to this report. 
Form PCT/IPEA/409 (Box I) (July 1998)* 
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inti:hnational i>hi:liminahy kxamination iii>:h)HT 



International application No. 
PCT/US00/i.'5S0l 



V. Reasoned statement under Article 35(2) with regard to novelt>s inventive step or industrial applicabiUt>". 
citations and explanations supporting such statement ^ 



1. stnteniout 

Novelty (N) 



Claims ^^-^^ 



Claims i-'^o 



Inventive Stop (IS) 



Industrial Applicability (I A) 



Claims ^^-^ ^ 



Claims i-^" 



Claims 
Claims 



1-51 



NONE 



YES 
NO 

YES 
NO 



YES 
NO 



2. citations and explanations (Rule 70.7) 

Claims \-^6 lack novelty under PCT Article 33(2) as being anticipated by Nayar (5,760.8^26). 

Reearding claim I. Nayar discloses an omnidirectional imaging apparatus 100. comprising receiving rays over a large 
field of view in a first direction (reflector 130 receives light, figure la. column 5. lines +5.65); a small .magmg w.dth m a second 
direction (image sensor 110 receives parrallel light rays 145 (figure la. column 5. lines 45-65); directing only said portion of 
said rays upon an area of an image sensor (figure la. column 5. lines 45-65). . / n 

Regarding claim <2, Nayar discloses wherein said directing includes interposing at least one mirror (reflector 135. 
fieure la)and at least one lens (lens ll!2. figure la). 

Regarding claim 3, Nayar discloses wherein said directing step distributes said portion of said rays upon the image 
sensor at points corresponding to a plurality of different radial directions of the large field of view (figure la). 
Regarding claim 4, Nayar discloses the step of interpolating (column 9. lines 5-<2Q). 
Regarding claim 5, Nayar discloses sampling of the interpolated data (column 9. lines 5-i20). 
Regarding claim 6, Nayar discloses the large field of view is approximately 360 degrees (figure lO). 
Regarding claim 7, Nayar disclo.ses the area of the imaging .sensor is an annular ring having a radial width ( figure 
5. column 7, line through column G. line iJO). 

Regarding claims 8. 11 and 15. Nayar discloses said directing step distributes .said portion of said rays throughout 
the radial width at points corresponding to a plurality of diflerent radial directions of the large field of view (figure 5. column 
7. line 6*2 through column G. line *20). 

Regarding claim 9. Nayar discloses the large field of view is approximately 180 degrees (figure la). 
Regarding claim 10. Nayar discloses the area of imaging sensor is a semi-circular annular portion having a radial 

width (figure 5). i cu \ 

Regarding claim liJ. Nayar discloses an omnidirectional (Continued on Supplemental bheet.) 
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Supplemental Box 

(To iM> us(h1 when the sp;u'c> in any of the pixH-oain^ U>xcs is not suiru iont) 



( ontiuiiatiou of: Boxes 1 - VIII 
V. 2. REASONED STATEMENTS - CITATIONS AND EXPLANATIONS {Continued): 

imaeine aooaratus acquiring image slice data at the plurality of known routional positions(figure 9a. column 9. me 55 
Zugh coTumnTo 1 ne 22); receiving rays over a large Held of view in a firs, direction (reflector 130 receives l.ght. figure 
la column 5 lir^es 45-65); a small imaging width in a second direction (image sensor 1 10 receives parrallel l.ght rays 145 
(figure U colln 5. lines 45-65); directing only said portion of said rays upon an area of an .mage sensor (figure la. column 
5. lines 45-65)^^^^^ ^^^^^ ^^^^^ ^^^^ of mapping the omnidirectional image to a display (figure la. figure 

'"'"Tegardirg Itl^ Nayar discloses the large field of view is approximately 360 degrees (figure 10); the parallel 
rays are dVrecSd on the image sensor in an amtular ring patttern having a radial width ( figure 5. column 7. l.ne 62 through 
column 6.^1ine 20).^ ^^^^ ^^^^^ ^^^^^^^^ ^^^^^ ^.^^^^.^^ orrhogot^l to the axis of roution and the parallel rays 

corresponding to a planar sheet (figures 8, 12, column 10. lines 44-53). . .u r H:,-n,i«„ «nH the narallel ravs 

Regarding claim 17. Nayar discloses the second direction is non-orthogonal to the first direction and the parallel rays 

corresponding to a conical sheet (figure 5). < linoc dS- 

Regarding claim 18. Nayar discloses conical mirror and telecentrtc lens arrangement (figure la. column 5. lines 45 

Regarding claim 19. Nayar discloses using a non-convex mirror in cooperation with an imaging lens (figure la. 

column 5, lines 45-63). 

Regarding claim 20. Nayar discloses the step of interpolating between .mages (figure 12). 

ReeardinR claim 21 Nayar discloses acquiring a plurality of overlapping image str.ps(figure 9a. column 9. line 55 
through coZn To. Iine"2) ideni^fying common features in overlapping regions (figure 12. step 1245. specifying 
Sumn S Tnes 4;-67); combining coLcutive image strips (figure 12. step 1260, interpolating step. - "-"Je columrfo 
bSng the combined images to form omnidirectional image dau (figure 12. step 1270, forming a d.g.tal .mage, column 10, 

lines 44-67.^^^^^_^^ ^^^^ ^ ^^^^^ ^^^^^^^ ^^^^.^^ omnidirectional image data to a display (figure la, column 9. lines 
Regarding claim 23, Nayar discloses said plurality of strips is at least 3 strips (figure la). 

Regarding claim 24, Nayar discloses an imaging sensor (image sensor 1 10. figure la. column 5. ' "^ 45-67) a least 
one mirror (feflectSr 135. figure la, column 5. lines 45-67); at least one lens (lens 1 12. figure -U^mn 5- 'mes 45-67). 

Regarding claim 25. Nayar discloses wherein at least one mirror is a conical mirror and said at least one lens is a 

telecentric lens arrangement (figure la). „a i.>nc i« an imaeine 

Regarding claim 26. Nayar discloses at least one mirror is a non-convex mirror and at least one lens is an imaging 

lens having n entrance pupil (figure la). 



Claims 27-37 lack an inventive step under PCT Article 33(3) as being obvious over Nayar (US 5.760.826) in view of Braun 
(US 5.532 737)^ ^ ^^^^ ^^.^^^^^^^ ^.^^^^^^ ^^^^^ „.irror^ However. Braun teaches a camera 

arrangement with wide field of view, which includes mirror 130 and mirror 160 (figure 3A. column 5. Imes 6-48). Therefore. 

haie ien obvious to one of ordinary skill in the art at the time the invention was -de to modify the device in 
Nayar by the teaching of Braun in order to avoid the iamge reversal problem (column 5. lines 45-48) 

Regarding claim 28. Braun discloses wherein the second mirror has a near focus and a far focus and wherem he 
first mirror section projects said portion of rays through the nera focus and the second mirror directs said portion of rays to the 
iamee sensor at a position proximate the far focus (figure 3A. column 5. lines 30-48). ^ u <■ <• . .c ic 

lamge s^n^^-^^*;^. ^^^^^ ^^^^,^^^3 f^„3 located in from of the first mirror section and the far focus is 

located behind the first mirror section (figure 3A). ^ • • .„i„c«iH mirmr Hnu,Pver it is a 

Regarding claim 30. Braun fails to specifically disclose the second mirror is an ellipsoid mirror. However, it is a 
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matter J^^'^'^"^.^^^^^ 3^ ^^^^^ ^^-^^ specifically disclose the first mirror is a conical mirror and the second mirror is a 
parabolic mirror. However, it is a matter of design choice. ... 

Regarding claim 32, Nayar disclsoes the iamge sensor is located behind an apex of the conical mirror (figure la). 

Regarding claim 33. Nayar discloses said portion of said rays are directed upon the imaging sensor at points 
corresponding to a plurality of differem radial directions of the large field of view (figure la). 

Regarding claim 34, Nayar discloses the large field of view is substamially 360 degrees (figure 10); the area of the 
imaging sensor in an annular ring having a radial width ( figure 5, column 7, line 62 through column 6 line 20). 

Regarding claims 35, 37, Nayar discloses wherein said rays are directed throughout the radial width at points 
corresponding to a plurality of differem radial directions of the large field of view (figure la). 

Regarding claim 36, Nayar discloses wherein the large field of view is asector less than 360 degrees(figure la), the 
image sensor is a section of an annular ring (figure la). 



Claims 38-51 meet the criteria set out in PCT Article 33(2).(4), because the prior art does not leach or fairly suggest an 
imaging system for acquiring an omnidirectional mosaic image, comprising a rotational drive system operat.vely coupled to the 
image slice camera, the rotational drive system rotating the image slice camera along an axis of rotation substantially parallel 
to the first direcction; a control unit operatively coupled to the rotational drive system and image slice camera, the control unit 
receiving the image slice data from the image slice camera and recording corresponding rotational positional information of the 
rotational drive system, the comrol unit acquiring a plurality of image slices at known rotational positions and concatenating 
said image slices to form an omnidirectional mosaic image data. 



NEW CITATIONS 

NONE 
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\. This written opinion is the ^""^^ 



, (first, etc.) drawn by this International Preliminary Examining Authority. 



2. This opinion contains indications relating to the following items: 



I 




II 


□ 


III 


□ 


IV 


□ 


V 




VI 


□ 


VII 


□ 


VIII 


□ 


The applicant 



citations and explanations supporting such statement 
Certain documents cited 

Certain defects in the international application 
Certain observations on the international application 
3, The applicant is hereby invited to reply to this opinion. 




How? 
Also 



See the time limit indicated above. The opjlieant may, \ 




By submitting a written reply, accompanied, where appropriate, by amendments, according to Rule 66.3. 
For the form and the language of the amendments, see Rules 66.8 and 66.9. 

For an additional opportunity to submit amendments, see Rule 66.4. 
For the examiner's obligation to consider amendments and/or arguments, see Rule 66.4 bis. 
For an informal communication with the examiner, see Rule 66.6. 
If no reply is filed, the international preliminary examination report will be established on the basis of this opinion. 

4. The final date by which the international preliminary 

examination report must be established according to Rule 69.2 is: 20 JANUARY 2002 
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Washington, D.C 30231 
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Authorized officer 
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I. Basis of the opinion 



1. With regard to the elements of the international application:* 
[x] the international application as originally filed 



the description: 
pages 
pages , 
pages 



1-31 



, as originally filed 



NONE 



filed with the demand 



NONE 



, filed with the letter of 



[x\ the claims: 

pages 

pages 

pages 

pages 



32-39 



NONE 



NONE 



, as originally filed 

. , as amended (together with any statement) under Article 19 

, filed with the demand 



NONE 



, filed with the letter of 



[3c] the drawings: 



pages 
pages 
pages 



1-22 



, as originally filed 



NONE 



, filed with the demand 



NONE 



I x| the sequence listing part of the description: 
pages NONE 



, filed with the letter of. 



pages 
pages 



NONE 



, as originally filed 

, filed with the demand 



NONE 



, filed with the letter of . 



2. With regard to the language, all tiie elements marked above were available or furnished to this AuthonQ^ in the language m which 
the international application was filed, unless otfierwise indicated under this item. . 
These elements wei« available or furnished to tfiis Authority in the following language wnicn is. 

I I the language of a translation furnished for the purposes of international search (under Rule 23.1(b)). 

I I the language of publication of the international application (under Rule 48.3(b)). 

|~| the language of the translation fiimished for the puiposes of international preliminaiy examination (under Rules 552 and/ 
or 553). 

3. With reg3id to any nucleotide and/or amino add sequence disclosed in the international appUcalion, the written opinion was 
drawn on tiie basis of the sequence listing: 

contained in the international application in printed form. 
I I filed together with the international application in computer readable form. 
I I furnished subsequently to this Authority in written form. 
I I furnished subsequently to this Authority in computer readable form. 

□ The statement that the subsequently furnished written sequence listing does not go beyond the disclosure in the 
international application as filed has been furnished, 

□ The statement that the information recorded in computer readable form is identical to the writen sequence listing has 
b^n furnished. 

4 [3 The amendments have resulted in the cancellation of: 

Q the description, oases NONE 

[El the claims, Nos. NONE 

He] the drawings, sheets/fig NONE 

5. Q This opinion has been drawn as if (some of) the amendments had not been made, since they have been considered to go 
b^ond the disclosure as filed, as' indicated in the Supplemental Box (Rule 70.2(c)). 

♦ Replacement sheets which have been Jumished to the receiving Office in response to an invitation under Article 14 are referred to 
in this opinion as "originally filed". 
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V. Reasoned statement under Rule 66.2{a){ii) with regard to novelty, inventive step or Industrial applicability; 
1 citations and explanations supporting such statement 


1. statement 






Novelty (N) 


rininifi 38-51 


YES 


PlnitnR 1-26 


NO 


1 Inventive Step (IS) 


nifliiiiR 38-51 


YES 


Claims ^-^"^ . 


NO 


1 Industrial Applicability (lA) 


ninims 1-51 


YES 


rininiR NONE 


NO 









2. citations and explanations 

Claims 1-26 lack novelty under POT Article 33(2) as being anticipated by Nayar (5.760.826). 

Regarding claim 1, Nayar discloses an omnidirectional imaging apparatus 100. comprising receiving rays over a large 
field of view in a first direction (refiector 130 receives light, figure la. column 5. lines 45-65); a small imaging width m a second 
direction (image sensor UO receives parrallel light rays 145 (figure la. column 5. lines 45^65); directmg only said portion of 
said rays upon an area of an image sensor (figure la, column 5. lines 45-65). / n . , «c 

Regarding claim 2. Nayar discloses wherein said directing includes interposing at least one mirror (reflector 135. 

figure la)and at least one lens (lens 112. figure la). . , • r a ^« tK« ;r«a<rp 

Regarding claim 3. Nayar discloses wherein said directing step distributes said portion of said rays upon the image 
sensor at points corresponding to a plurality of different radial directions of the large field of view (figure la). 
Regarding claim 4. Nayar discloses the step of interpolating (column 9. lines 5-20). 
Regarding claim 5. Nayar discloses sampling of the interpolated data (column 9. lines 5-20). 
Regarding claim 6. Nayar discloses the large field of view is approximately 360 degrees (figure 10). 
Regarding claim 7. Nayar discloses the area of the imaging sensor is an annular ring having a radial width ( figure 

5. column 7, line 62 through column 6. line 20). • r a ♦Kr«..*rhm.r 

Regarding claims 8. 11 and 15. Nayar discloses said directing step distributes said portion of said rays throughout 
the radial width at points corresponding to a plurality of different radial directions of the large field of view (figure 5. column 

7, line 62 through column 6. line 20). /r.„..^ io\ 

Regarding claim 9. Nayar discloses the large field of view is approximately 180 degrees (figure la). 
Regarding claim 10. Nayar discloses the area of imaging sensor is a seminrircular annular portion having a radial 

width (figure 5). . . o i i cu • \ 

Regarding claim 12. Nayar discloses an omnidirectional (Continued on Supplemental Sheet.) 
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TIME UMTT: 

The time limit set for response to a Written Opinion may not be extended. 37 CFR 1.484(d). Any response 
received after the expiration of the time limit set in the Written Opinion will not be considered in preparing the International 
Preliminary Examination Report. 

V. 2. REASONED STATEMENTS - CITATIONS AND EXPLANATIONS (Continued): 

imaging apparatus, acquiring image slice dau at the plurality of known rotational positions(figure 9a, column 9, hne 55 
through column 10. line 22); receiving rays over a large field of view in a first direction (reflector 130 receives light, figure 
la. column 5, lines 45-65); a small imaging width in a second direction (image sensor 110 receives parrallel light rays 145 
(figure la, column 5, lines 45-65): directing only said portion of said rays upon an area of an image sensor (figure la. column 

5. lines 45-65). ^ 4r 

Regarding claim 13, Nayar discloses the step of mapping the omnidirectional image to a display (figure la, tigure 

12, column 10, line 54-64). u n i 

Regarding claim 14. Nayar discloses the large field of view is approximately 360 degrees (figure 10); the parallel 
rays are directed on the image sensor in an annular ring patttern having a radial width ( figure 5. column 7, line 62 through 

column 6. line 20). ^ • j u n i 

Regarding claim 16. Nayar discloses the second direction is orthogonal to the axis of rotation and the parallel rays 

corresponding to a planar sheet (figures 8. 12, column 10, lines 44-53). 

Regarding claim 17, Nayar discloses the second direction is non-orthogonal to the first direction and the parallel rays 

corresponding to a conical sheet (figure 5). , c i- 

Regarding claim 18. Nayar discloses conical mirror and telecentric lens arrangement (figure la, colunm 5, lines 43- 

63) 

Regarding claim 19. Nayar discloses using a non-convex mirror in cooperation with an imaging lens (figure la. 
column 5. lines 45-63). 

Regarding claim 20. Nayar discloses the step of interpolating between images (figure 12). 

Regarding claim 21. Nayar discloses acquiring a plurality of overlapping image strips(figure 9a, column 9, line 55 
through column 10. line 22); identifying common feamres in overlapping regions (figure 12. step 1245. specifying i™ge view, 
column 10, lines 44-67); combining consecutive image strips (figure 12. step 1260. interpolating step, column 10. lines 44-67); 
blending the combined images to form omnidirectional image daU (figure 12. step 1270. forming a digital image, column 10. 
lines 44-67. 

Regarding claim 22. Nayar discloses mapping the omnidirectional image data to a display (figure la. column 9, lines 

^^^^^ Regarding claim 23. Nayar discloses said plurality of strips is at least 3 strips (figure la). 

Regarding claim 24, Nayar discloses an imaging sensor (image sensor 110. figure la. column 5. lines 45-67); at least 
one mirror (reflector 135. figure la, column 5. lines 45-67); at least one lens (lens 112, figure la, column 5, lines 45-67). 

Regarding claim 25. Nayar discloses wherein at least one mirror is a conical mirror and said at least one lens is a 
telecentric lens arrangement (figure la). • • • 

Regarding claim 26, Nayar discloses at least one mirror is a non-convex mirror and at least one lens is an imaging 

lens having n entrance pupil (figure la). 



Claims 27-37 lack an invemive step under PCT Article 33(3) as being obvious over Nayar (US 5,760.826) in view of Braun 
(US 5 532 737) 

Regarding claim 27. Nayar fails to specifically disclose first and second mirror. However. Braun teaches a camera 
arrangement with wide field of view, which includes mirror 130 and mirror 160 (figure 3A. column 5, lines 6-48). Therefore, 
it would have been obvious to one of ordinary skill in the art at the time the invention was made to modify the device in 
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Nayar by the teaching of Braun in order to avoid the iamge reversal problem (column 5, lines 45-48). 

Regarding claim 28, Braun discloses wherein the second mirror has a near focus and a far focus and wherein the 
first mirror section projects said portion of rays through the nera focus and the second mirror directs said portion of rays to the 
iamge sensor at a position proximate the far focus (figure 3A» column 5, lines 30-48). 

Regarding claim 29, Braun discloses the near focus is located in front of the first mirror section and the far focus is 
located behind the first mirror section (figure 3A). 

Regarding claim 30, Braun fails to specifically disclose the second mirror is an ellipsoid mirror. However, it is a 
matter of design choice. 

Regarding claim 31. Braun fails to specifically disclose the first mirror is a conical mirror and the second mirror is a 
parabolic mirror. However, it is a matter of design choice. 

Regarding claim 32, Nayar disclsoes the iamge sensor is located behind an apex of the conical mirror (figure la). 

Regarding claim 33, Nayar discloses said portion of said rays are directed upon die imaging sensor at points 
corresponding to a plurality of different radial directions of the large field of view (figure la). 

Regarding claim 34, Nayar discloses the large field of view is subsUntially 360 degrees (figure 10); the area of the 
imaging sensor in an annular ring having a radial width ( figure 5. column 7. line 62 through column 6, line 20). 

Regarding claims 35, 37. Nayar discloses wherein said rays are directed throughout Uie radial width at points 
corresponding to a plurality of different radial directions of the large field of view (figure la). 

Regarding claim 36, Nayar discloses wherein the large field of view is asector less than 360 degrees(figure la), the 
image sensor is a section of an annular ring (figure la). 



Claims 38-51 meet the criteria set out in PCT Article 33(2)-(4). because the prior art does not teach or fairly suggest an 
imagmg system for acquiring an omnidirectional mosaic image, comprising a rotational drive system operatively coupled to the 
image slice camera, the routional drive system rotating the image slice camera along an axis of rotation substantially parallel 
to the first direcction; a control unit operatively coupled to the rotational drive system and image slice camera, the control unit 
receiving the image slice data from the image slice camera and recording corresponding rotational positional information of the 
rotational drive system, the control unit acquiring a plurality of image slices at known routional positions and concatenating 
said image slices to form an omnidirectional mosaic image data. 



NEW CITATIONS 



NONE 
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FOR FURTHER see Notification of Transmittal of Intemationul Search Report 
ACTION (Form rcT/ISA/220) as well as, where applicable, item 5 below. 


International application No. 
PCT/USOO/25801 


International filing date (day/month/year) 
20 SEPTEMBER 2000 


(Earliest) Priority Date (day/numth/year) 
20 SEPTEMBER 1999 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK 



This international search report has been prepared by this International Searching Authority and is transmitted to the applicant 
according to Article 18. A copy is being transmitted to the International Bureau. 

This international search report consists of a total of f sheets. 

I x| It is also accompanied by a copy of each prior art document cited in this report. 



1. Basis of the report 

a. With regard to the language, the international search was carried out on the basis of the intemati<mal application in the 

language in which it was filed, unless otherwise indicated under this item. 

□ the international search was carried out on the basis of a translation of the international application furnished to this 
Authority (Rule 23.1(b)). 

b. With regard to any nucleotide and/or amino acid sequence disclosed in the inteniatioDal £q>p]icatioa, the international seaidi 
was carried out on the basis of (he sequence listing: 

I I contained in the international application in written form. 

I I filed together with the international application in computer readable form. 

I I furnished subsequently to this Authority in written form, 

I I furnished subsequently to this Authority in computer readable form. 

I 1 the statement that the subsequently furnished written sequence listing docs not go beyond the disclosure in the 

^ — ' international application as filed has been furnished. 

I I the statement that the infoimaiion recorded m computer readable form is identical to the written sequence listing has been 
furnished. 

2- I I Certain claims were found unsearchable (See Box I). 

3. Unity of invention is lacking (See Box II). 

4. With regard to the title, 

I x| the text is approved as submitted by the applicant. 

j — j the text has been established by this Authority to read as follows: 

5. With regard to the abstract, 

I I the text is approved as submitted by the applicant. 

Px| the text has been established, according to Rule 38.2(b), by this Authority as it appears in 
Box III. The applicant may, within one month from the date of mailing of this international 
search report, submit comments to this Authority. 

6. The figure of the drawings to be published with the abstract is Figure No. 



I I as suggested by the applicant. 

I X| because the applicant failed to suggest a figure. 

I I because this figure better characterizes the invention. 



I I None of the figures. 
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Box III TEXT OF THE ABSTRACT (Continuation of item 5 of the first siteet) 



NEW ABSTRACT 



Systems and methods for generating an omnidirectional mosaic image are 
presented in which a nimiber of images are acquired about an axis of rotational. 
The images have a large field of view along the axis of rotation and a small field of 
view or image width in a second direction. The images can be image strips, 
formed from non-parallel rays directed onto an image sensor (1(X)8), which are 
formed from a narrow width of parallel rays directed onto imaging sensor (1008). 
The images are combined to form a spherical mosaic. In the case of overlapping 
image strips, image combination can be performed by identifying common features 
in the overlapping regions and aligning consecutive image strips accordingly. A 
blending algorithm can then be used to improve image fidelity in the overlapping 
regions. 
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II. New International Application 




Earliest priority date 
(Day/Nlonth/Year) 



SCREENING DISCLOSURE INFORMATION: In order to assist in screening the accompanying international 
application for purposes of determining whether a license for foreign transmittal should and could be granted and for 
other purposes, the following information is supplied. (Note: check as many boxes as apply): 

A* n invention disclosed was not made in the United States. 

B. There is no prior U.S. application relating to this invention. 

C. ^] The following prior U.S. application(s) contain subject matter which is related to the invention disclosed in the attached 

international application. (NOTE: priority to these applications may or may not be claimed on form PCT/RO/20J 
(Request) and this listing does not constitute a claim for priority). 



application no. 




filed on 




application no. 




filed on 





D. □ 

E. □ 



application(s) identified in paragraph C. 

The present international application |^ contains additional subject matter not fou nd in the prior U.S. application(s) 
identified in paragraph C. above. The additional subject matter is found on pages | 

and □ DOES NOT ALTER □ MIGHT BE CONSIDERED TO ALTER the general nature of the invention in a 
manner which would require the U.S. application to have been made available for inspection by the appropriate defense 
agencies under 35 U.S.C. 1 8 1 and 37 CFR 5. 1 . See 37 CFR 5.15 



III. Q A Response to an Inyitation from the ROAJS. The following document(s) is (are) enclosed: 
A. Q A Request for An Extension of Time to File a Response 
A Power of Attorney (General or Regular) 
Replacement pages: 



B. □ 

c. □ 



pages 




of the request (PCT/RO/101) 


pages 




of the figures 


pages 




of the description 


pages 




of the abstract 


pages 




of the claims 





D. n Submission of Priority Documents 



Priority document 



Priority document 



E- n Fees as specified on attached Fee Calculation sheet form PCT/RO/101 annex 



IV. LJ A Request for Rectification under PCT 91 



TJ 



A Petition 



IT 



A Sequence Listing Diskette 



V. Kl Other (please specify): Demand for International Preliminary Examination (3 sheets). Fee Calculation Sheet, a Communication, a 

postcard and a check In the amount of $627. 



1 1 Applicant 


Paul D. Ackenman 


1^ Attorney/Agent (Reg. No.) 
39.891 


^ ^ Typ^ name of signer 


1 1 Common Representative 


C — ^ — 

Signature 



The person 
signing this 
form is the: 



PTO-I382 (Rev. 4-1995) 



Copyright 1996 Legalsoft 



U.S. Department of Commerce: Patent and Trademark Office 



THIS PAGE BLANK (uspto) 



The demand must be filed directly wfit^^^ competent International Preliminary Examining ''^j^ 'O' or, if two or more Authorities are 
with the one chosen by the applicant. jhe full name or two-letter code of that Authority may be indicated by the applicant on the line 

IPEA/ US 



PCX 

DEMAND 

under Article 31 of the Patent Cooperation Treaty: 
The undersigned requests that the international application specified below be the subject of 
international preliminary examination according to the Patent Cooperation Treaty and 
hereby elects all eligible States (except where otherwise indicated). 



CHAPTER II 



For International Preliminary Examining Authority use only 



Identification of IPEA 


Date of receipt of DEMAND 


Box No. I IDENTIFICATION OF THE INTERNATIONAL APPLICATION 


Applicant's or agent's file reference 
32709-PCT 


International application No. 
PCT/USOO/25801 


Intemational filing date (day/month/year) 
20 September 2000 ( 20.09.00 ) 


(Earliest) Priority date (day/month/year) 
20 September 1999 ( 20.09.99 ) 



Title of invention 

SYSTEMS AND METHODS FOR GENERATING SPHERICAL MOSAIC IMAGES 



Box No. II APPLICANT(S) 



Name and address: (Family name followed by given name: for a legal entity, full official 
designation. The address must include postal code and name of country.) 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK 
1 1 6th Street and Broadway 
New York, NY 10027 
US 


Telephone No.: 


Facsimile No.: 


Teleprinter No,: 


State (that is. country) of nationality: 
US 


State (thai is, country) of residence: 
US 



Name and address: (Family name followed by given name: for a legal entity, full official designation. The address must include postal code and 
name of country.) 



NAYAR. SHREE K. 

560 Riverside Drive, Apt. 15D 

New York. NY 10027 

US 



State (that is, country) of nationality: 




State (that is, country) of residence: 


IN 




US 



Name and address: (Family name followed by given name: for a legal entity, full official designation. The address must include postal code and 
name of country ) 



KARMARKAR, AMRUTA 
1916 Cambridge Court, Apt. 2B 
Palatine. IL 60074 
US 



State (that is. country) of nationality: 
IN 



State (that is, country) of residence: 
US 



I I Further applicants are indicated on a continuation sheet. 



Form PCT/lPEA/401 (first sheet) (July 1998; reprint January 2001) 



Legaistar 2001 . Form PCTOEM See Notes to the demand form 



THIS PAGE BLANK (uspto) 



Sheet No. 



International application No. 

PCT/USOO/25801 



Box No. Ill AGENT OR COMMON REPRESENTATIVE; OR ADDRESS FOR CORRESPONDENCE 



The following person is (Xl agent [H] common representative 

and has been appointed earlier and represents the applicant(s) also for international preliminary examination. 

CH is hereby appointed and any earlier appointment of (an) agent(s) /common representative is hereby revoked. 

□ is hereby appointed, specifically for the procedure before the International Preliminary Examining Authority, in 
addition to the agent(s)/common representative appointed earlier. . 



Name and address: (Family name followed by given name; for a legal entity, full official 
The address must include postal code and name of country.) 



TANG, HENRY and 

ACKERMAN. PAUL D. 

Baker Botts LLP 

30 Rockefeller Plaza 

New York. NY 10112-0228 

US 



Telephone No.: 
(212) 705-5000 



Facsimile No.: 
(212) 705-5020 



Teleprinter No.: 



□ Address for correspondence: Mark this check-box where no agent or common representative is/has been appointed and 
the space above is used instead to indicate a special address to which correspondence should be sent. 



Box No. IV BASIS FOR INTERNATIONAL PRELIMINARY EXAMINATION 



Statement concerning amendments:* 
1. The applicant wishes the international preliminary examination to start on the basis of: 
IXI the international application as originally filed. 



the description 



the claims 



the drawings 



□ 
□ 

□ 
□ 
□ 

□ 
□ 



as originally filed 

as amended under Article 34 

as originally filed 

as amended under Article 19 (together with any accompanying statement) 
as amended under Article 34 

as originally filed 

as amended under Article 34 



2. [Z] The applicant wishes any amendment to the claims under Article 1 9 to be considered as reversed. 

3 rn The applicant wishes the start of the international preliminary examination to be postponed until the expiration of 

20 months from the priority date unless the International Preliminary Examing Authority receives a copy of any 

amendments made under Article 19 or a notice from the applicant that he does not wish to make such amendments 
(Rule 69. 1 (d)). (This check-box may be marked only where the time limit under Article 19 has not yet expired.) 
* Where no check-box is marked, international preliminary examination will start on the basis of the international application as 
originally filed or, where a copy of amendments to the claims under Article 19 and/or amendments of the international 
application under Article 34 are received by the International Preliminary Examining Authority before it has begun to draw up 
a written opinion or the international preliminary examination report, as so amended. 



Language for the purposes of international preliminary examination: ^^n9[ish 

I XI which is the language in which the international application was filed. 

I I which is the language of a translation furnished for the purposes of international search. 

I I which is the language of publication of the international application. 

I I which is the language of the translation (to be) furnished for the purposes of international preliminary examination. 
Box No. V ELECTION OF STATES 



The applicant hereby elects ail eligible States (that is, all States which have been designated and which are bound by Chapter II of the 

PCT) 

excluding the following States which the applicant wishes not to elect: 



Form PCT/IPEA/401 (second sheet) (July 1998; reprint January 2001) 



LegalStar 2001. Form PCTOEM 



See Notes to the demand form 



TH\S PAGE BUMK 



Sheet No. 



International application No. 
PCT/USOO/25801 



Box No. VI CHECKLIST 



The demand is accompanied by the following elements, in the language referred to in 
Box No. IV, for the purposes of international preliminary examination: 



1 . translation of international application 

2. amendments under Article 34 

3 copy (or where required, translation) 
of amendments under Article 1 9 



4. copy (or, where required, translation) 
of statement under Article 1 9 



5. letter 

6. other (specify) 



sheets 
sheets 

sheets 

sheets 
sheets 
sheets 



For International Preliminary 
Examining Authority use only 

received not received 



□ 
□ 

□ 

□ 
□ 



□ 
□ 

□ 

□ 

□ 
□ 



The demand is also accompanied by the item(s) marked below: 

1. fee calculation sheet 

2. I I separate signed power of attorney 

3 I I copy of general power of attorney; 
I— I reference number, if any: 



4. I I statement explaining lack of signature 

^ I I nucleotide and or amino acid sequence listing in 
I — « computer readable form 

other (specify): Transmittal Letter and 

Communication 



Box No. Vn SIGNATURE OF APPLICANT, AGENT OR COMMON REPRESENTATIVE 



Next to each signature, indicate the name of the person signing and the capacity in which the person signs (if such capacity is not 
obvious from reading the demand). 




For International Preliminary Examining Authority use only ' 



1 . Date of actual receipt of DEMAND: 



2- Adjusted date of receipt of demand due 
to CORRECTIONS under Rule 60.1(b): 



3 I — I The date of receipt of the demand is AFTER the expiration of 1 9 months I | The applicant has been 

I — I from the priority date and item 4 or 5, below, does not apply. ■ — ' mformed accordmgly. 



4 I — I The date of receipt of the demand is WITHIN the period of 1 9 months from the priority date as extended by virtue of 
I — I Rule 80.5. ■ 



5 . I — I Although the date of receipt of the demand is after the expiration of 19 months from the priority date, the delay in arrival is 
I — I EXCUSED pursuant to Rule 82. 



For International Bureau use only 



Demand received from IPEA on: 



Form PCT/IPEA/401 (last sheet) (July 1998; reprint January 2001) 



LegalStar 2001. Form PCTDEM ^^^^^ demand form 



THIS PAGE Bimn msim 



PCX 



CHAPTER II 



FEE CALCULATION SHEET 
Annex to the Demand for international preliminary examination 

- For International Preliminary Examining Authority use only 



International 
application No. 



PCT/USOO/25801 



Applicant's or agent's 
file reference 



32709-PCT 



Date stamp of the IPEA 



Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK 



Calculation of prescribed fees 



1 . Preliminary examination fee 

2. Handling fee (Applicants from certain States are 
entitled to a reduction of 75% of the handling fee. 
Where the applicant is (or ail applicants are) so 
entitled, the amount to be entered at H is 25% of the 
handling fee.) ■ • • 

3. Total of prescribed fees 

Add the amounts entered at P and H 

and enter total in the TOTAL box 




Mode of Payment 






1 — 1 authorization to charge deposit 
1 1 account with the IPEA (see below) 


□ 


cash 


cheque 


□ 


revenue stamps 


1 1 postal money order 


□ 


coupons 


1 1 bank draft 


□ 


other (specify): 



Deposit Account Authorization (this mode of payment may not be available at all IPEAs) 

The IPEA/ US | | is hereby authorized to charge the total fees indicated above to my deposit account. 

(this check-box may be marked only if the conditions for deposit accounts of the IPEA so permit) is 
hereby authorized to charge any deficiency or credit any overpayment m the total fees mdicated 
above to my deposit account. 



02-4377 



18 April 2001 



Deposit Account Number 




Date (day/month/year) 



Signature 



Form PCT/IPEA/401 (Annex) (July 1998; reprint January 2001) Legaistar 2001. Form pctdfee See Notes to the fee calculation sheet 
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PATENT COOPERATION TREATY 

From the INTERNATIONAL SEARCHING AUTHORITY 



To: HENRY TANG 

BAKER BOTTS LLP 

30 ROCKEFELLER PLAZA 

NEW YORK NY 10112-0228 



BAKfTR DOTTS L.L.R 
0iFEB27 PM 2:59 

prT TO 



NOTIFICATION OF TRANSMITTAL OF 
THE INTERNATIONAL SEARCH REPORT' 
OR THE DECLARATION 



(POT Rule 44.1) 




Date of Mailing 
(day/month/year) 



22 FEB 2001 



Applicant's or agent's file reference 

32709-PCT 



FOR FURTHER ACTION See paragraphs 1 and 4 below 



International application No. 
PCTAJSOO/25801 



International filing date 
(day/month/year) 

20 SEPTEMBER 2000 



Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK 



1. 1 x| The applicant is hereby notified that the international search report has been established and is transmitted herewith. 
Filing of amendments and statement under Article 19: 

The applicant is entided, if he so wishes, to amend the claims of the international application (see Rule 46): 

When? The time limit for filing such amendments is normally 2 months from the date of transmittal of the 
international search report; however, for more details, see the notes on the accompanying sheet. 

Where? Directly to the International Bureau of WIPO Docketed 
34, chemin des Colombettes 



1211 Geneva 20, Switzerland 
Facsimile No.: (41-22)740.14.35 

For more detailed instructions, see the notes on the accompanying sheet. 



For 



j /?Z^2001B 




The applicant is hereby notified that no international search report will be established and that the declaration under 
Article 17(2Xa) to that effect is transmitted herewith. 

With regard to the protest against payment of (an) additional fee(s) under Rule 40.2. the applicant is notified that: 

□ the protest together with the decision thereon has been transmitted to the International Bureau together with the 
applicant's request to forward the texts of both the protest and the decision thereon to the designated Offices. 

I I no decision has been made yet on the protest, the applicant will be notified as soon as a decision is made. 



4. Further action(s): The applicant is reminded of the following: 

Shortly after 18 months from the priority date, the international application will be published by the International Bureau. If 
the applicant wishes to avoid or postpone publication, a notice of withdrawal of the international application, or of the 
priority claim, must reach the International Bureau as provided in niles 90 bis 1 and 90 bis 3, respectively, before the 
completion of the technical preparations for international publication. 

Within 19 months from the priority date, a demand for international preliminary examination must be filed if the applicant 
wishes to postpone the entry into the national phase until 30 months from the priority date (in some Offices even later). 

Within 20 months from the priority date, the applicant must perform the prescribed acts for entry into the national phase before 
all designated OlTices which have not been elected in the demand or in a later election within 19 months from the priority 
date or could not be elected because they are not bound by Chapter II. 







Name and mailing address of the ISA/US 

Commissioner of Patents and Trademarks 

Box PCT 

Washington, D.C. 20231 
Facsimile No. (703) 305-3230 


Authorized officer / / 

LUONGNGUYEN^^|/[^ y i 

Telephone No. (703) 308-9297 


m 


r 



Form PCT/lSA/220 (July 1998)* 



(See notes on accompanying sheet) 



NOTES TO FORM PCT/ISA/220 (continued) 
iS^^XlSI r/in'''" •««<'•«-», must be expUlned In ,hc 

1. (^cre origimlly there were 48 claims and after amendment of come claims there are 51 1- 

Qaims 1 to 29, 31. 32, 34, 35. 37 to 4fi replaced by amended claims bearioe the sam/n.t,K-- 
claims 30. 33 and 36 unchanged; new claiins 49 to 51 added." ^ ' numbers; 

2. [Where originally there were X5 claims and after amendment of ill claims there are 1 1 1- 
-aaims 1 to 15 replaced by amended claims 1 to 1 1." 

^' K^dSiSp'"'"^'*'^" "*='«''«"<''^«>=«><l»cntsooQ,i,tinanoclling,omeclaims.ndin.ddiag 

12*!™* i •<> * •a<J J 4 unchanged; claims 7 to 13 canoelled; new cUims 15. 16 and 17 >d<lM< - 
-Oaims 7 to U cancelled: new claims 15, 16 and 17 added; .llothercU^ imSfn^ 
4. fWbeie various lands of amendments are made]: 

Qaiins l-lOundungedjcUims lltol3,18and WcaaocIIed; cUima 14, 15«nd 16reDl«cedbv.m«,x-^ 
clami 14; claim 17 subdivided into uneoded claims 15. 16 a^ 17; new daims 20^ 21 



"Sutcmcat under Article 19(1)" (Role 46,4) 

The amendments may be aocompuiied by a cUI'Sieat explaining the amendmeats tad indicating anv imn«,/ 
J5;id^9Sr^^** °° description «.d the Srawings (which canaS bf ."Sf 

The tutemeat will be published with the iatetnational applicatioa and the amended dainu. 
The autemeat should be brief, it should not exceed 500 words if ia Eaglisb or if aanslated iato English. 
It should aot be confounded with and does not rq>Uae the letter iadicitiag the difXetcaces between the d.im* 
asCled and as amended. It must be Gled oa a iepatate sheet and murtbekS^ m h«dS? 
ptefetably by tisiag the wotds "Suteaient imder /ffide 19(1) - "emuiea as suca by a headmg. 



Itihouldiiotooiiuiiijmy disM^ gevcfa report orthcrelcvtnocof duiioos 

Kwt. Refereace lo attuoas, relevtat to t given claim, ooauined in the iatematiocul seart± 



oooCiincd in that rcporL .^^^^^ma-v^ w ^^mmv^u^ remrvu uj « given init o 
report may be made only in coonectioQ with an amendment of that cUim- 



la what language ? 

l^A^^'^tf^^.'^'^^ ^ """^ Unguagc in which the intcmatiooal tpplicaUon is published, th^ letter 

and anv lUtcmcat aooocnpanjong the amendments muat be ia the «mc languid aa the inti^tiooal tppIicaUoa 
,f that language » Enghst or French; otherwise, it must be in English or?r«Si, at the ^ Sfplfant 

Coaseqocncc If a dcaaod for Intcnudoaal prellmlaary cxamtoaCkM haa already been died ? 

h'Jl^^^^^'lSU^^ l?' • iotemati anal preliminary examination 

J^^^S^^^J^T^^ •pplicanl must preferably, a: the aame time of Gling & tmexid^enti with the 
SSSri??^^ ^ latematiooal Preliminary Examining 

Cooacqococ* wMh regard to tnoaUdoo of the loC«maCio«al appUcatiott for entry Into the nadoiia] phase ? 

The applicants attention is drawn to the fact that, whcxe upon entry into the national phase, a translation of the 
daina as amended under Article 19 may have to be furnished to the designated/elccSo&ocs, inst^d otTr 
to addiuon lo, the translation of the daims as filed. uau^u «^ wr 

^r further deuiU on the requirements of eadi designated/elected OfBce, see Volume Hof the PCT Applicant's 



Notes to Fora PCT/lSA/220 (icoood »boet) (July 1992) 
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